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fidelity  and  convenience  of  use. 
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SUMMARY 


Objective 

The  objective  was  to  conduct  an  empirical  cost-effectiveness  evaluation  of  three  training  devices  within  the  6883 
Ccnverter/i1  light  Control  Test  Station  portion  of  an  Avionics  course  at  Lowry  AFB.  '’’hese  devices  were  (a)  the  6883  test 
stution  actual  equipment  trainer,  (b)  a  high  fidelity,  three-dimensional  simulator,  and  (c)  a  low  fidelity,  flat  panel 
simulator. 


Background 

Maintenance  of  electronic  military  equipment  is  a  critical  component  of  preparedness.  With  the  continued  los8  of 
experienced  technicians  from  the  Armed  Services  and  the  increased  dependence  on  more  junior,  less  experienced 
technicians  to  maintain  electronic  equipment,  training  has  become  a  critical  issue.  Traditionally,  hands-on  training  in 
the  maintenance  of  the  Converter/Flight  Control  System  for  that  F-l  1 1  aircraft  has  been  provided  to  Avionics  Maintenance 
trainees  using  operational  6883  test  station  equipment.  Yet,  there  are  many  problems  with  current  practices  of 
maintenance  training.  In  particular,  it  is  generally  not  possible  to  insert  malfunctions  into  operational  equipment  for 
the  purpose  of  teaching  or  practicing  troubleshooting  skills.  Additionally,  operational  availability  of  actual  equipment 
trainers,  consistency  of  training,  scope  of  training,  safety,  equipment  reliability,  and  cost  are  additional  limitations  of 
actual  equipment  trainers. 


Approach 

At  Lowry  AFB,  129  F-l  11  Avionics  Maintenance  trainees  in  the  6883  Automatic  Test  Station  block  of  training 
were  randomly  assigned  to  one  of  three  training  devices:  (a)  actual  6883  equipment  trainer,  (b)  high  fidelity  three- 
dimensional  6883  simulator,  or  (c)  low  fidelity  (flat  panel)  6883  simulator.  The  training  syllabus  was  not  changed,  and 
simulator  training  was  limited  to  these  malfunctions  that  could  be  presented  on  all  three  devices. 

A  specially  developed  “hands-on”  troubleshooting  performance  test  (TPT)  was  administered  immediately  after 
training  in  lieu  of  more  traditional  instructor  subjective  ratings  of  student  performance.  A  paper-and-pencil 
troubleshooting  test  of  complex  switching  problems  was  given  at  the  end  of  the  training  block.  Student  and  instructor 
attitudinal  data  pertaining  to  the  use  of  simulators  and  actual  equipment  in  training  were  obtained.  Follow-up  measures 
after  6883  training  included  performance  in  subsequent  blocks  of  training  and  job  proficiency  ratings  by  field  supervisors. 
The  respective  costs  for  each  of  the  three  training  devices  were  analyzed  and  compared  for  a  projected  15-year  life  cycle 
of  the  training  block. 


Specifics 

Student  performances  were  essentially  equal  following  training  either  on  simulators  or  on  actual  equipment.  Average 
hands-on  student  performance  test  scores  were  (a)  actual  equipment  85%,  (b)  high  fidelity  88%,  and  (c)  low  fidelity 
86%.  Paper-and-pencil  troubleshooting  test  scores  in  all  three  groups  averaged  40%.  The  simulators  were  more  reliable 
in  delivering  training.  Training  device  availability  rates  were  (a)  actual  equipment  60%,  (b)  high  fidelity  73%,  and  (c) 
low  fidelity  93%.  The  15-year  life-cycle  cost  comparisons  favored  simulators,  and  the  low  fidelity  simulator  in  particular, 
(a)  actual  equipment  $5.3M,  (b)  high  fidelity  82. 1M,  and  (c)  low  fidelity  I1.6M.  User  acceptance  of  simulators  was 
generally  favorable. 


Conclusions/Recommendations 

When  actual  test  equipment  and  specific  simulators  of  that  equipment  were  used  in  a  traditional  instructional  course, 
no  substantial  student  performance  differences  were  found  during  or  after  the  training.  The  traditional  use  of  such  training 
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equipment  permitted  only  2  days  exposure  for  each  group  with  respective  devices,  Individualized  self-paced  practice 
was  precluded.  Even  under  such  restricted  conditions,  the  more  economical  simulators  produced  equivalent  student 
performance  when  compared  to  the  actual  equipment.  In  this  traditional  training  setting,  all  students  were  deficient  in 
handling  complex  switching  problems  as  measured  on  the  paper-and-pencil  test  at  the  completion  of  training.  Simulators 
were  decidedly  less  expensive  to  procure  and  operate,  were  more  readily  available,  and  in  general  were  favorably 
accepted  for  the  training. 

Applications  should  more  fully  utilize  the  potential  of  maintenance  simulators  to  increase  student  hands-on  contact 
and  provide  more  in-depth  malfunction  training,  as  well  as  providing  greater  opportunities  for  self-paced  individualized 
practice. 
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PREFACE 


This  project  was  conducted  for  the  Air  Force  Human  Resources 
Laboratory,  Brooks  AFB.  This  evaluation  was  conducted  under  the  tech¬ 
nical  monitorship  of  Dr.  Gerard  M.  Deignan,  Air  Force  Human  Resources 
Laboratory,  Project  Scientist  for  Program  2361-02-01.  Dr.  Edgar 
Smith  is  the  Simulation  Program  Manager. 

This  Final  Report  completes  Phase  III  of  the  cost  and  training 
effectiveness  evaluation  of  the  6883  3-D  and  2-D  simulators  as 
compared  to  the  6883  AET,  for  training  intermediate  level  P—1 1 1 
avionics  maintenance  personnel  at  Lowry  AFB.  The  plan  for  this 
evaluation  was  discussed  in  detail  in  the  1979  Phase  I  Interim  Report 
(AFHHL-TR-79-13)  and  a  comparison  between  the  3-D  simulator  and  the 
6883  AET  was  presented  in  the  Phase  II  Report  (AFHRL-TR-80-24) . 

The  evaluation  outlined  in  this  report  was  developed  and 
implemented  by  the  Social  Systems  Research  and  Evaluation  Division  of 
the  Denver  Research  Institute,  University  of  Denver,  Denver,  Colorado, 
under  Contract  Number  F33615-78-C-0018.  Dr.  Louis  F.  Cicchinelli  was 
the  Principal  Investigator  and  overall  Project  Director. 
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I.  INTRODUCTION 


Maintenance  of  electronic  military  equipment  is  a  critical 
component  of  preparedness.  With  the  continued  loss  of  experienced 
technicians  from  the  Armed  Services  and  the  increased  dependence  on 
the  more  junior,  less  experienced  technicians  to  maintain  electronic 
equipment,  training  has  become  an  increasingly  important  concern 
(Fallows,  1981;  Shriver  &  Hart,  1973)*  Traditionally,  hands-on 
training  in  the  maintenance  of  the  F-111  Converter/Flight  Control 
System  has  been  provided  to  avionics  maintenance  trainees  using  an 
operational  6883  AET.  In  fact,  the  use  of  AETs  has  been  standard 
procedure  throughout  the  Air  Force  aircraft  maintenance  training 
schools.  Yet  there  are  many  problems  with  this  current  approach  to 
maintenance  training  including  availability  of  AETs,  consistency  of 
training,  scope  of  training,  safety,  equipment  reliability  and  cost. 

In  addition,  the  AETs  are  not  designed  to  be  training  devices; 
therefore,  they  do  not  withstand  the  wear  and  tear  of  constant  use  by 
inexperienced  operators  and  do  not  necessarily  provide  the  best 
training  experience  (Beavers,  1980;  Deignan  &  Cicchinelli,  1980; 

Folley  &  Elliott,  1967;  Malec,  1980;  Miller,  1980;  Orlansky  &  String, 
1977,  1981a;  Shriver,  Fink,  &  Trexler,  1964;  Shriver  &  Foley,  1974; 
Siegel,  Bergman,  Federman,  &  Sellman,  1972;  Spangenberg,  1974,  1976). 
In  particular,  it  is  generally  not  possible  to  insert  malfunctions 
into  operational  equipment  for  the  purpose  of  teaching  troubleshooting 
skills. 


Technician  preparedness  for  field  duty  is  based  primarily  on 
maintenance  training  received  at  ATC  schools.  Limitations  in  training 
(Engel  &  Rehder,  1970;  Orlansky  &  String,  1 98 1b)  have  directed 
attention  to  the  use  of  simulated  training  devices  to  deliver 
electronic  maintenance  training.  The  application  of  simulation 
techniques  and  equipment  to  technical  training  is  not  a  new  concept. 
However,  only  in  the  past  few  years  have  extensive  efforts  been 
undertaken  to  carefully  and  systematically  investigate  the  use  of 
computer-based  simulators  for  electronic  maintenance  training.  These 
efforts  have  resulted  in  several  important  conclusions: 

•  Maintenance  training  simulators  (MTSs)  can  be  cost 
effective  (Deignan  &  Cicchinelli,  1980;  Eggemeier  & 

Klein,  1981;  Fink  &  Shriver,  1978;  Fink,  Shriver, 

Downing,  and  Miller,  1978;  Montmerlo,  1977;  Orlansky 
&  String,  1 98 1  a ;  Vestewig  &  Eggemeier,  1981; 

Wheaton,  Rose,  Fingerman,  &  Leonard,  1976) 

•  MTSs  can  provide  training  comparable  to  AETs 
(Cicchinelli,  Harmon,  Keller,  &  Kotten3tette,  1980; 

Daniels,  Datta,  Gardner,  &  Modrick,  1975;  Orlansky  & 

String,  1981a;  Wright  &  Campbell,  1975) 


•  MTSs  may  provide  the  best  instructional  approach  for 
tasks  requiring  problem  solving  skills  and  cognitive 
strategies  (Brown,  Burton,  Bell,  &  Bobrow,  1974; 

Deignan  A  Cicohinelli,  1980;  Kearsley,  1977;  Mallory 
&  Elliott,  1978;  Spangenberg,  1974,  1976) 

•  MTSs  can  deliver  part-task  training  to  which  the  AET 
is  not  highly  amenable  (Folley,  Joyce,  Mallory,  & 

Thomas,  1971a,  1971b;  Miller,  1980;  Siegel  et  al., 

1972) 

While  only  a  few  simulator  maintenance  training  programs  have 
been  in  operation  long  enough  to  assess  their  long-range  impacts,  some 
common  problems  associated  with  the  use  of  training  simulators  have 
become  apparent.  First,  there  is  the  need  to  assess  the  degree  to 
which  the  fidelity  of  the  training  device  will  allow  establishment  of 
equivalent  training  objectives  and  contexts  (Mallory  &  Elliott,  1978). 
Criterion-based  measures  of  training  effectiveness  are  important  in 
this  context  of  training  assessment.  Previous  researchers  have  had 
difficulty  in  establishing  such  measures  without  incurring  high  costs 
(Deignan  &  Cicchinelli,  1980;  Siegel  et  al.,  1972).  Second,  it  is 
critical  that  the  manner  in  which  the  simulator  will  be  incorporated 
and  used  in  the  existing  training  protocol  be  determined  in  advance. 
Without  such  a  plan,  it  is  unlikely  that  simulators  designed  to  be  an 
integral  component  of  training  will  be  used  as  such.  And  third, 
problems  can  arise  as  a  result  of  instructor  opposition.  That  is, 
instructors  may  consider  their  teaching  role  to  be  threatened  by  the 
incorporation  of  simulation  methods,  or  they  may  disagree  with 
required  changes  in  the  academic  structure  (Miller,  1980).  These 
three  issues  are  more  fully  discussed  in  the  following  sections. 


Fidelity 


One  key  consideration  in  the  design,  application  and 
evaluation  of  most  simulated  training  devices  is  the  degree  of 
fidelity  of  the  simulator  to  the  AET.  This  issue  is  important  due  to 
the  assumption  by  instructors  that  only  an  AET  (or  a  highly  similar 
unit)  can  provide  the  necessary  hands-on  training,  and  to  the  belief 
that  the  training  environment  must  continue  to  be  highly  comparable  to 
the  field  environment  (Miller,  1980;  Winthrop,  1981).  The  focus  may 
be  on  physical  fidelity,  which  is  defined  as  the  comprehensiveness  and 
level  of  detail  with  which  the  real  world  (AET)  is  physically 
represented  (Narva,  1978),  and/or  psychological  fidelity,  which  is 
defined  as  the  degree  to  which  the  trainee  perceives  the  simulated 
device  as  a  duplicate  of  the  AET  (Foley,  1963;  Miller  4  Gardner, 

1975). 
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Despite  attempts  to  incorporate  realism  into  the  training 
environment,  some  differences  between  training  and  field  environments 
usually  exist.  In  the  classroom,  there  are  often  limitations  on  the 
number  and  types  of  tasks  which  can  be  taught.  Artificial  conditions 
associated  with  training  environments  are  often  uncontrolled;  and 
emotional  and  attitudinal  elements  of  the  field  environment  are 
difficult  to  duplicate  in  the  classroom. 

For  these  reasons,  Miller  (1980)  and  Mallory  and  Elliott 
(1978)  have  suggested  that  the  techniques  used  in  training  must  relate 
to  the  context  of  the  training,  and  that  fidelity  of  simulated 
training  devices  should  be  based  on  the  behavioral  cues  associated 
with  task  performance.  Opportunities  for  technicians  to  make  errors 
or  perform  inadequately  must  be  equally  represented  in  simulated 
training  as  in  real  world  conditions.  It  has  been  suggested  that 
part-task  simulation  and  increased  physical  fidelity  of  maintenance 
simulation,  requiring  the  simulation  of  the  internal  functions  of  the 
original  equipment  and  not  just  the  outward  functions,  can  address 
these  deficiencies  (Miller,  1980;  Winthrop,  1981). 

More  importantly,  there  have  been  difficulties  associated  with 
simulation  design.  For  example,  there  has  been  a  tendency  for 
simulated  training  tasks  to  be  less  difficult  than  real-world  tasks 
(Cicchinelli  et  al.,  1980;  Mallory  &  Elliott,  1978),  thus  detracting 
from  the  overall  level  of  skills  training  achieved.  On  the  other 
hand,  Fink  and  Shriver  (1978)  maintain  that  only  task-specific 
components  of  training  equipment  should  be  simulated  and  that  they 
should  have  a  high  degree  of  psychological  fidelity.  Displays  and 
controls  beyond  those  required  for  selected  maintenance  tasks  may  be 
irrelevant  or  even  distracting  to  the  novice  technician.  In  fact,  too 
much  fidelity  of  equipment  may  retard  learning  of  the  critical 
knowledge  or  skills  to  be  acquired  (Deignan  &  Cicchinelli,  1980; 

Dwyer,  1971,  1975). 


Role  of  the  Simulator 


A  second  important  issue  related  to  the  use  of  simulated 
training  devices  is  the  role  of  this  equipment  in  the  training 
program.  Such  devices  can  be  used  to  complement  the  existing  training 
on  the  AET  or  to  replace  the  AET  as  a  training  instrument.  Fink  and 
Shriver  (1978)  suggest  that  a  simulated  trainer  is  most  useful  prior 
to  contact  with  the  AET.  Actual  equipment  should  be  reserved  for  on- 
the-job  training  (OJT)  whereas  the  simulator  should  be  used  in  the 
classroom  to  teach  logical  skills.  In  addition,  it  is  argued  that 
significant  cost  reductions  can  be  obtained  through  the  substitution 
of  simulators  for  expensive  and  frequently  unavailable  AETs  (Beavers, 
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1980;  Wheaton  et  al.,  1976;  Winthrop,  1981).  An  alternative 
viewpoint,  that  simulators  should  be  used  as  complements  to  AETs,  is 
presented  by  Miller  (1980).  He  contends  there  is  a  need  for  both 
training  devices:  simulators  to  deliver  primary  skill  acquisition  and 
AETs  for  the  final  integration  of  discrete  skills  (see  also  Fleishman, 
1975). 


User  Acceptance 


The  use  of  simulated  trainers  can  be  expected  to  meet  with 
stiff  opposition  from  the  user  community,  primarily  training 
instructors  (Fink  &  Shriver,  1978;  Pieper,  1969).  The  primary  reason 
for  this  resistance  is,  of  course,  the  presumption  that  the  training 
environment  is  and  must  remain  identical  to  the  field  environment. 
Further,  there  is  the  tendency  to  judge  new  training  approaches 
against  the  existing  curricula  which  have  been  accepted  as  standards. 
If  change  is  to  be  accepted,  it  is  clear  that  innovative  training 
curricula  must  be  accompanied  by  innovations  in  personnel  training  and 
support  areas  (Miller,  1980).  These  required  changes  in  attitude  and 
perceptions  can  be  realized  more  easily  if  the  benefits  of  simulated 
trainers  (e.g.,  flexibility  of  training,  safety  to  personnel,  rapid 
adaptation  to  technological  change,  reduced  need  for  large  inventories 
of  parts)  are  explained  more  completely  or  experienced  directly. 

In  this  report,  the  results  of  the  comparative  analysis  of  the 
Converter /Flight  Control  Test  Station,  a  3-D  simulator  and  a  2-D 
simulator  are  presented.  This  analysis  includes  performance  and  cost 
comparisons,  and  concludes  with  a  discussion  of  uses  of  simulators  and 
trainers  for  avionics  maintenance  training. 
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II.  EVALUATION  DESIGN 


This  ohapter  provides  an  outline  of  the  major  components  of 
the  Phase  III  evaluation  design  and  research  objectives.  This 
information  is  necessary  to  place  the  discussion  of  the  evaluation 
environment  presented  in  Chapter  III  in  a  proper  context. 


Research  Objectives 


The  objective  of  this  study  was  to  design  and  implement  a  com¬ 
prehensive  cost  and  training  effectiveness  evaluation  of  a  Converter/ 
Flight  Control  (6883)  ?-D  simulator  and  a  6883  2-D  simulator,  as 
compared  to  the  use  of  an  operational  6883  AET  in  training  intermediate 
level  (I-level)  F-111  avionics  maintenance  personnel. 

The  evaluation  plan  outlined  in  this  section  was  developed  as 
Phase  III  of  an  ongoing  evaluation  effort.  The  design  was  developed 
to  be  consistent  with  that  used  in  Phase  II  to  compare  the  training 
and  cost  effectiveness  of  the  6883  3-D  simulator  and  the  AET 
(Cicchinelli,  1979;  Cicchinelli  et  al.,  1980).  While  the  general 
framework  remained  consistent  with  the  proposed  statement  of  work, 
some  modifications  were  made  in  the  number  of  experimental  groups 
required  to  collect  adequate  data  for  the  assessment.  Additionally, 
some  changes  were  made  in  the  data  collection  instruments  based  on 
experience  acquired  during  Phase  II  of  the  project.  New  preassessment 
instruments  were  also  added  in  this  phase  to  enable  a  comparison  of 
groups  on  pre-existing  abilities.  The  overall  design  was  divided  into 
three  major  components:  classroom  performance,  field  performance,  and 
cost  analysis. 


Classroom  Performance 


The  basic  design  actually  used  to  assess  student  classroom 
performance  as  a  function  of  training  mode  differed  substantially  from 
the  one  originally  proposed  for  this  Phase  III  study.  Both  designs 
are  discussed  in  this  section. 


Original  Phase  III  Design 

The  Phase  II  evaluation  plan  to  assess  classroom  performance 
was  revised  for  Phase  III.  Groups  E  and  F  in  Figure  1  represent  the 
experimental  cells  added  to  the  design  employed  in  Phase  II  of  this 
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effort  (Cicchinelli  et  al.,  1980).  These  additions  were  expected  to 
make  possible  an  examination  of  performance  differences  in  student 
performance  as  a  function  of  the  training  equipment  used.  Further,  it 
would  have  been  possible  to  identify  the  extent  to  which  the  use  of 
the  2-D  simulator  for  training  would  affect  performance  on  the  actual 
equipment  by  comparing  the  test  scores  of  groups  E  and  F. 


Testing  Mode 


3-D  2-D 

Training  Mode  AET  Simulator  Simulator 

AET 

3-D  Simulator 
2-D  Simulator 

Figure  1 .  Research  design  originally  proposed  for  Phase  III. 


A 

B 
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C 

D 

E 
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In  addition  to  incorporating  these  new  groups  into  the  design, 
the  amount  of  training  on  the  6883  test  station  was  expected  to  be 
increased  from  about  3  days  (Phase  II  level)  to  6  days.  This  planned 
"increase"  was  more  accurately  described  as  a  return  to  the  level  of 
training  used  before  the  evaluation  began  in  February  1979.  Not 
surprisingly,  the  Phase  II  experience  indicated  that  2  days  of 
training  on  a  specific  test  station  were  probably  insufficient  to 
observe  differential  impacts  of  training  mode  on  student  performance. 
Thus,  it  was  proposed  to  collect  Phase  III  student  data  from  airmen 
trained  for  6  days  on  one  of  the  equipment  trainers.  All  students 
would  then  be  tested  on  the  actual  6883  equipment.  The  groups 
proposed  are  shown  in  Figure  2. 


Training  Mode 
AET 

3-D  Simulator 
2-D  Simulator 


Test  Mode 


Figure  2. 


Research  design  originally  proposed  for  assessing 
performance  differences  under  a  6  day  training  regimen. 


Despite  the  attempt  to  maintain  consistency  between  Phase  II 
and  Phase  III  data,  subsequent  training  format  changes  precluded  any 
direct  comparison  of  student  performance  measures.  First,  the 
training  period  for  the  Converter/Flight  Control  block  was  extended  to 
9  days  by  combining  the  theory  and  practical  portions  of  training. 
Revised  training  requirements  included  an  additional  15  hours  of 
diagnostic  testing  on  the  6883  Test  Station.  This  time  represents  the 
practical  or  hands-on  portion  of  the  training  block,  as  compared  to  a 
theory  portion  which  is  taught  primarily  by  use  of  technical  orders 
(TOs),  chalkboard,  and  classroom  pencil-and-paper  exercises.  Second, 
each  training  device  (i.e.,  AET,  3-D  simulator,  2-D  simulator)  was 
scheduled  to  be  used  in  varying  amounts  depending  on  its  capability  of 
providing  specific  Test  Replaceable  Unit  (TRU)  and  Line  Replaceable 
Unit  (LRU)  training. 


Phase  III  Design  Employed 

These  course  format  and  training  time  changes  were  implemented 
in  February  1980,  after  Phase  II  data  collection  was  completed.  Due 
to  these  changes,  the  proposed  evaluation  methodology  was  modified  to 
include  re-collection  of  data  from  3-D  simulator-trained  students  and 
AET-trained  students.  Data  collection  from  2-D  simulator- trained 
students  was  planned  to  begin  several  months  later  when  the  2-D 
simulator  was  scheduled  to  be  available  for  student  training.  These 
changes  in  format  and  methodology  necessitated  a  new  evaluation  design 
which  again  included  the  collection  of  performance  data  on  students 
trained  on  the  AET  and  the  3-D  simulator.  The  final  research  design 
is  shown  in  Figure  3. 


Testing  Mode 

3-D  2-D 

Training  Mode  AET  Simulator  Simulator 

AET 

3-D  Simulator 
2-D  Simulator 

Figure  3*  Phase  III  research  design  implemented. 


•This  experimental  group  (B)  was  included  in  the  Phase  II 
study  but  eliminated  in  the  Phase  III  plan. 
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It  should  be  noted  that  whereas  2-D  and  AET.  training  groups 
received  all  6883  instruction  on  a  single  trainer,  3-D  classes  always 
had  some  limited  experience  with  either  the  AET  or  2-D  devices  for 
patch  panel  training.  While  this  factor  was  tracked  closely  throughout 
the  project  period,  it  did  not  emerge  as  a  significant  experimentation 
consideration  for  final  analyses. 

Each  student  was  assigned  to  one  of  the  five  experimental 
groups.  It  was  planned  that  an  equal  number  of  students  would  be 
assigned  to  the  three  training  modes  with  AET  testing  as  shown  in 
Figure  3  (A,  C,  E).  Further,  an  equal  number  of  students  would  be 
tested  on  the  same  equipment  on  which  they  were  trained  (A,  D,  F).  In 
this  manner,  it  would  be  possi* 'e  to  examine  performance  differences 
as  a  function  of  the  training  equipment  used,  while  controlling  for 
the  possible  effects  of  negative  interference  due  to  using  different 
training  and  testing  equipment  for  a  single  group. 

The  evaluation  plan  for  assessing  classroom  performance  also 
included  administration  of  preassessment  measures  related  to  reading 
and  technical  vocabularies,  figure  recognition,  and  logical  reasoning 
ability.  Previous  analysis  has  shown  that  neither  scores  from  the  Air 
Force  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  nor  previous 
block  scores  were  adequate  predictors  of  differences  in  student 
performance  (Cicchinelli  et  al.,  1980).  Thus,  the  results  of  the 
preassessment  instruments  were  expected  to  describe  more  completely 
the  student  population,  and  to  provide  statistical  control  for 
possible  group  differences  resulting  from  pre-existing  student 
abilities. 


Job  Performance 


The  design  shown  in  Figure  4  was  used  to  assess  the  impact  of 
training  mode  on  job  performance  in  the  field.  The  design  includes  12 
experimental  groups  and  is  sensitive  to  the  possibility  that  testing 
classroom  performance  itself  constitutes  additional  training.  Thus, 
the  assessment  of  field  performance  must  be  conducted  in  view  of  the 
levels  of  training  resulting  from  the  various  combinations  of  classroom 
training  and  testing  modes. 

It  should  be  noted  that  this  design  illustrates  all  the 
theoretical  combinations  of  training/testing  modes  and  field  assign¬ 
ments  of  interest.  However,  since  field  assignments  are  dictated  by 
the  need  for  specific  personnel,  it  was  not  expected  that  an  equal 
number  of  students  would  be  in  each  cell,  or  even  that  all  cells  would 
be  filled. 


]  8 


Cost  Analysis 


The  comparison  of  costs  associated  with  using  the  6883  AET, 
the  6883  3-D  simulator  and  the  6883  2-D  simulator  for  training  was 
based  on  the  model  shown  in  Figure  5.  The  model  consists  of  a  matrix 
of  six  major  cost  categories  and  two  components  of  life  cycle  costs. 
The  model  is  simply  the  "ingredients  approach"  discussed  by  Levin 
(1975)  in  which  cost  elements  are  identified  and  evaluated  consistent 
with  the  ATC  acquisition  and  training  environment.  The  cost  elements 
associated  with  each  cost  category  are  evaluated  either  as  Investment 
Costs  (one-time  costs)  or  as  Recurring  Annual  Costs,  consistent  with 
the  AFHRL  perspective  on  economic  analysis  (Department  of  Defense, 
n.d. ;  Williams,  1977). 


Life  Cycle  Costs 


Cost  Categories1 


Investment  Year  Operating  Years  1  to  15 


Facilities 

Equipment 

Instructional 

Materials/Training 

Personnel 

Students 

Miscellaneous 


The  model  used  to  establish  the  life  cycle  cost  comparison 
also  provides  the  framework  for  evaluating  alternative  simulator 
implementation  strategies,  particularly  those  involving  proposed 
changes  in  such  factors  as  student  flow,  course  length,  and  time  to 
complete  specific  instructional  blocks.  In  this  sense,  the  model  is 
general. 


To  assess  cost  effectiveness,  the  three  training  devices  were 
assumed  to  provide  equal  training  effectiveness.  Given  this  assumption, 
the  life  cycle  cost  comparison  among  trainers  indicates  which  trainer 
exhibits  the  least  total  cost  of  ownership.  This  is  a  useful  approach 
since  it  establishes  baseline  cost  data  and  is  consistent  with  the 
original  simulator  design  objective  of  developing  a  functional  replace¬ 
ment  for  the  6883  AET  (Miller  &  Gardner,  t975).  The  cost  analysis 
discussion  addresses  the  validity  of  the  equal  effectiveness  assumption 
in  light  of  factors  (e.g.,  equipment  availability,  Specialty  Training 
Standards  [STS])  which  were  found  to  influence  training  effectiveness. 


Hypotheses  to  be  Tested 


This  evaluation  plan  was  designed  to  address  the  following 
hypotheses: 

•  Practical  training  on  the  6883  3-D  and  2-D 
simulators  and  the  6883  AET  results  in  equivalent 
performance  on  the  standard  ATC  block  tests  for 
subsequent  training. 

•  Airmen  trained  on  the  6883  3-D  and  2-D  simulators 
and  the  AET  are  equally  accurate  in  solving  trouble¬ 
shooting  problems. 

•  Airmen  trained  on  the  6883  3-D  and  2-D  simulators 
and  the  AET  are  equally  efficient  in  solving 
troubleshooting  problems. 

•  Airmen  trained  on  the  6883  3-D  and  2-D  simulators 
and  the  AET  operate  the  AET  with  equal  proficiency. 

•  Airmen  trained  on  the  6883  3-D  and  2-D  simulators 
and  the  AET  are  equally  familiar  and  comfortable  in 
operating  the  AET  without  supervision. 

•  Airmen  trained  on  the  6883  3-D  and  2-D  simulators 
and  AET  will  acquire  equivalent  job-related 
experience. 
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•  Airmen  trained  on  the  6883  3-D  and  2-D  simulators 
and  the  AET  will  be  equally  capable  of  operating  the 
AET  in  the  field. 

•  Airmen  trained  on  the  6883  3-D  and  2-D  simulators 
and  the  AET  are  equally  capable  of  operating  assigned 
test  stations,  other  than  the  6883  station,  in  the 
field. 
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III.  THE  EVALUATION  ENVIRONMENT 


Throughout  the  project  period,  an  ongoing  analysis  of  the  F- 
111  avionics  maintenance  training  and  field  environments  was  conducted. 
This  assessment  included  the  collection  a^.d  review  of  course-related 
documents,  irterviews  with  instructors  and  students,  and  direct  observa¬ 
tion  of  the  6883  classroom  proceedings  as  well  as  other  test  station 
training  blocks.  Due  to  changes  in  several  elements  of  training  during 
the  project  period,  it  was  necessary  periodically  to  review  and 
reanalyze  course-related  activities.  The  review  of  the  classroom  and 
field  environments  identified  a  number  of  factors  which  had  a  direct 
impact  on  the  implementation  and  management  of  the  evaluation  effort. 

The  most  important  of  these  factors  were: 

•  training  objectives 

•  training  program  format 

•  operational  status  of  the  6883  3-D  and  2-D  simulators 

•  reliability  of  training  equipment 

•  availability  of  equipment  cost  data 

•  student  assignment  and  flow 

•  assignment  and  OJT 

The  impact  of  each  of  these  factors  on  the  evaluation  is  discussed  in 
this  chapter  following  a  review  of  the  equipment  configurations  and 
uses. 


The  6883  Converter /Flight  Control  AET 


The  AET  is  a  large  (96"  x  72"  x  30")  unit  consisting  of  four 
interlocked  bays  (see  Figure  6).  Each  bay  consists  of  removable 
drawers,  or  TRUs,  which  allow  the  test  station  to  function  in  a  manner 
simulating  the  operation  of  the  F-111  aircraft.  To  this  AET  are 
attached,  through  cables  and  adapters,  LRUs  which  are  electronic  black 
boxes  removed  from  aircraft  because  of  suspected  malfunctions.  The 
AET,  by  simulating  the  aircraft  electronic  operation,  permits  trouble¬ 
shooting  of  the  LRU  in  a  workshop  environment.  One  of  the  key 
troubleshooting  components  is  the  switching  complex,  including  the 
patch  panel.  Comprised  of  1632  pins,  the  patch  panel  permits  tech¬ 
nicians  to  pinpoint  malfunctions  in  the  electronic  flow  between  the 
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TRUs  and  the  LRUs.  It  is  this  AET  which  has  served  as  the  model  for 
the  development  of  the  3-D  and  2-D  simulators. 


The  6883  3-D  Simulator 


The  3-D  simulator  was  designed  and  developed  to  appear  and 
operate  as  the  6883  AET  while  achieving  a  consistency  of  training 
(high  equipment  reliability  with  standardized  malfunction  simulations), 
an  increased  level  of  safety,  and  reduced  noise  levels.  The  3-D 
simulator  lacks  an  operational  patch  panel,  due  to  the  costs  estimated 
for  developing  a  high  fidelity  unit. 


The  final  configuration  of  the  6883  3-D  simulator  (Figure  7) 
includes  (a)  two  computers  (one  a  master,  the  second  a  slave  which 
directly  powers  the  simulator);  (b)  an  instructor  console  for  both 
monitoring  student  performance  and  inputting  original  programming;  (c) 
a  student  console  with  a  slide  projector  and  screen  (for  providing 
information  in  addition  to  the  technical  order  information)  and  a 
keyboard  for  student  responses  to  questions  displayed  on  a  CRT;  (d)  a 
four  bay,  3-D  stand  alone  trainer  with  limited  internal  wiring  of 
selected  TRUs;  (e)  mock-up  LRUs  with  simulated  cables  and  adapters; 
and  (f)  a  printer  for  recording  student  performance.  All  of  these 
components  are  required  in  normal  training  conditions. 


The  6883  2-D  Simulator 


The  2-D  trainer  is  configured  differently  from  the  AET  or  6883 
3-D  simulator  (Figure  8).  It  consists  of  (a)  a  master  console  with 
keyboard  and  slide  screen;  (b)  a  student  action  panel  on  which 
students  can  identify  an  action  to  be  taken  upon  a  defective 
component;  (c)  a  hard  copy  printer  for  monitoring  student  performance; 
(d)  a  flat  panel  mock-up  of  the  6883  AET;  and  (e)  an  LRU  panel 
(located  on  the  work  tray  of  bay  4)  which  simulates  the  actions  of  all 
relevant  6883  LRUs  in  one  unit  and  uses  simplified  adapters  and 
cables.  In  addition,  the  2-D  simulator  system  includes  three  part- 
task  trainers,  peripheral  to  the  flat  panel  components,  which  provide 
training  on  an  oscilloscope,  a  patch  panel  and  the  logic  of  the 
electronics  system.  These  three  part-task  trainers  can  be  used 
independently  of  the  flat  panel  trainer.  The  2-D  simulator  was 
designed  to  have  less  physical  fidelity  to  the  AET  than  the  3-D  unit 
but  greater  operational  fidelity.  That  is,  while  there  is  a 
noticeable  lack  of  detail  in  certain  2-D  components  such  as  cables  and 
LRUs,  the  2-D  device  does  provide  complete  patch  panel  training. 
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Figure  8.  The  6883  2-D  simulator. 


Training  Objectives 


Training  on  the  6883  Converter/Flight  Control  Systems  Test 
Station  occurs  as  part  of  a  23-week  intermediate  level  F-111  avionics 
maintenance  course.  The  objectives  of  this  course,  the  training 
format  and  the  associated  STS,  have  been  in  a  continuous  state  of 
change. 


Most  recently,  a  number  of  factors  have  contributed  to  the 
need  to  modify  course  objectives  and  content.  Perhaps  the  most 
important  factor  is  the  evolution  of  the  F-111  aircraft  itself.  As 
more  sophisticated  F-111  models  have  been  developed,  course  content 
has  been  modified  to  include  instruction  in  the  operation  and 
maintenance  of  new  test  stations  capable  of  testing  the  new  aircraft 
systems.  Prior  to  1978,  students  were  trained  as  test  station 
operators  (course  ABR326X1B;  9  days  of  6883  theory  and  practical 
training)  or  as  test  station  maintenance  personnel  (course  ABR326XOB; 

5  days  of  6883  theory  and  practical  training).  In  1978,  these  career 
options  were  integrated  into  a  single  career  path,  and  a  new  STS  was 
developed  to  reflect  course  modifications.  In  this  combined  course 
(Interim  Course  ABR326X1D),  theory  and  practical  training  on  the  6883 
test  station  were  reduced  to  a  total  of  8  days.  More  recently,  still 
another  Plan  of  Instruction  (POI)  was  developed  in  accordance  with  STS 
ABR326X4A  which  became  effective  in  April  1979.  This  course  includes 
expansion  of  skills  required  for  3-level  training. 

The  specifications  for  the  6883  3-D  simulator  were  developed 
when  operation  and  maintenance  training  were  alternative  career 
choices.  On  the  other  hand,  specifications  for  the  6883  2-D  simulator 
were  developed  during  the  Interim  course  (ABR326X1F)  which  combined 
the  operator  and  maintainer  components.  The  approach  used  to  analyze 
course  documentation  and  the  use  of  the  6883  actual  test  station  in 
training,  then,  was  to  compare  training  objectives  at  three  discrete 
points  in  time.  Specifically,  the  comparison  was  among  instructional 
objectives  used  when  (a)  maintenance  and  operation  of  the  6883  test 
station  were  separate  courses,  (b)  instructional  objectives  of  the 
1978  combined  Interim  course  (ABR326X1F)  were  in  place,  and  (c)  the 
training  objectives  used  for  course  ABR326X4A  were  in  place,  which  was 
during  this  evaluation  effort.  Briefly  stated,  the  analysis  of 
training  objectives  revealed  that  the  impact  of  these  changes  in 
objectives  on  the  evaluation  effort  could  be  isolated.  While  many 
objectives  remained  the  same  over  time,  some  objectives  (e.g., 
training  on  the  Yaw  Computer  LRU)  that  were  used  to  define  the 
simulator  capabilities  were  eliminated  due  to  the  significant 
reduction  in  training  time  available.  Since  these  training  exercises 
were  no  longer  used,  it  was  not  appropriate  to  include  the  associated 
simulator  capabilities  in  any  tests  of  student  performance,  and  the 
evaluation  design  reflects  this  decision. 
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While  course  objectives  did  change  over  time,  the  relevant  STS 
requirements  did  not  substantially  change.  This  finding  suggested 
that  the  specialty  standards  are  general  enough  to  allow  for  inter¬ 
pretation,  depending  on  the  perspective  of  the  instructional  staff. 

That  is,  training  and  field  personnel  could  easily  assume  that  somewhat 
different  skills  are  associated  with  specific  requirements,  such  as 
"troubleshooting."  The  impact  of  this  variability  on  the  evaluation 
effort  was  more  significant  than  any  specific  changes  which  took  place 
in  objectives.  Clearly,  the  lack  of  specific  criteria  for  adequate 
performance  poses  potential  problems  for  an  evaluation  that  attempts 
to  assess  training  effectiveness. 

A  portable  troubleshooting  device  to  be  used  as  an  independent 
performance  test  both  at  the  end  of  training  and  during  field  follow 
up  was  considered,  but  not  developed  due  to  budgetary  constraints. 

The  problem  was  circumvented  in  this  study  by  considering  only 
comparative  training  effectiveness  and  ignoring  the  more  basic  and 
possibly  more  important  issue  of  training  adequacy.  This  study 
addressed  the  question,  "How  do  simulator  trained  students  perform  as 
compared  to  AET  trained  students?"  rather  than  "Does  simulator  training 
adequately  prepare  technicians  for  field  duty?"  Although  an  assessment 
of  the  overall  adequacy  of  training  was  considered  beyond  the  scope  of 
this  project,  it  is  highly  recommended  that  "adequacy  of  training"  be 
a  primary  consideration  in  the  development  of  Air  Force  training  policy 
and  in  future  simulator  evaluation  efforts. 


Training  Program  Format 


The  comparative  analysis  of  course  content  over  the  project 
period  indicated  that  most  of  the  objectives  of  the  former  maintenance 
and  operations  courses  had  been  retained,  although  the  presentation 
had  been  greatly  modified  in  time  allocation  and  format.  It  was 
originally  proposed  to  rely  heavily  on  existing  test  instruments  to 
collect  relevant  data.  Further,  it  was  proposed  to  emphasize  training 
on  the  6883  2-D  simulator  since  previous  research  (see  Cicchinelli  et 
al.,  1980)  on  the  AET  and  6883  3-D  simulator  equipment  would  be  avail¬ 
able  to  serve  as  baseline  data.  However,  changes  in  the  6883  block 
test  instruments  and  changes  in  both  training  time  and  format  (after 
completion  of  the  Phase  II  AET  and  3-D  simulator  study  but  prior  to 
initiation  of  the  2-D  simulator  study)  precluded  use  of  the  previous 
work  as  baseline  data.  In  short,  there  were  no  standard  Air  Force 
test  instruments  nor  previous  comparative  evaluation  data  available 
that  could  be  used  to  compare  the  performance  of  a  baseline  group  of 
trainees  with  those  trained  on  either  the  6883  3-D  or  2-D  simulators. 


Prior  to  the  Phase  III  data  collection  period,  theory  and 
practical  training  on  the  6883  test  station  (5  days  and  3  days,  respec¬ 
tively)  were  considered  separate  blocks  of  instruction.  However,  in 
an  effort  to  take  advantage  of  the  training  equipment  as  a  visual 
complement  of  the  instructional  material,  the  practical  portion  of  the 
training  program  was  combined  with  the  theory  portion  to  form  a  9  day 
6883  instructional  block.  Hands-on  use  of  the  AET,  or  the  3-D 
simulator  or  2-D  simulator,  was  directly  incorporated  into  the  theory 
instruction.  The  6883  Block  POI  in  effect  during  the  study  period 
included  27  hours  of  practical  instruction,  although  instructors  esti¬ 
mate  that  approximately  3  days  of  practical  experience  is  provided  to 
each  class. 


Availability  of  Cost  Data 


The  essential  question  explored  in  the  Cost  Analysis  effort 
concerns  the  costs  of  ownership  and  maintenance  of  alternative  training 
devices.  As  might  be  expected,  some  difficulty  in  establishing  costs 
of  ownership  was  related  to  data  access  and  reliability.  Disaggregated 
estimates  of  the  development  costs  of  the  simulator  hardware,  software, 
and  courseware  available  from  the  manufacturers  are  not  releasable  for 
publication.  Thus,  for  the  purposes  of  this  report,  the  acquisition 
costs  of  simulator  hardware,  software,  and  courseware  have  been  lumped 
together  and  treated  as  "sunk"  costs.  These  sunk  costs  have  been 
included  in  the  assessment  of  the  life  cycle  costs  of  the  respective 
systems  as  Investment  Costs  (see  Chapter  V). 

The  contracted  cost  for  developing  and  manufacturing  the  6883 
simulators  does  not  reflect  the  total  expenditure  for  development  and 
production  of  the  devices  because  corporate  and  independent  research 
and  development  (R&D)  funds  were  used  in  years  prior  to  the  contract 
awards  to  develop  certain  aspects  of  the  technology  used.  Estimates 
of  the  dollar  value  of  these  somewhat  indirect  costs  were  not  avail¬ 
able  from  the  manufacturers.  The  manufacturers  believe  that  the 
actual  costs  incurred  in  development  and  production  of  the  first  6883 
simulators  are  not  useful  as  an  estimate  of  the  costs  of  producing 
additional  simulators.  For  example,  the  3-D  6883  simulator  was 
considered  to  be  a  research  device  incorporating  features  inconsistent 
with  production  models  and  current  computer  technology.  In  the  case 
of  the  3-D  simulator,  the  control  system  was  designed  to  be  expandable 
so  that  a  minimum  of  four  satellite  test  station  simulators  comparable 
to  the  6883  could  be  operated  simultaneously  from  the  same  instructor 
station.  Thus,  it  was  considered  important  to  be  able  to  estimate  the 
incremental  cost  of  hardware  and  software  development  associated  with 
the  additional  computer  that  provides  master  control  for  simultaneous 
operation  of  the  satellite  simulators.  However,  such  detailed  cost 
data  were  not  available  from  the  manufacturer. 


30 


Finally,  difficulty  was  encountered  in  regard  to  estimating 
the  AET  costs.  ATC  maintains  two  Burroughs  D84  computer  processors 
and  related  peripherals  to  actually  "run"  automatic  fault  isolation 
tests  on  10  test  stations  (including  the  6883)  used  in  F-111  training. 
Developing  life  cycle  cost  information  on  these  Central  Processor  and 
Controller  (CENPAC)  computers  for  purposes  of  allocating  costs  to  the 
operation  of  the  6883  test  station  was  seen  to  be  a  project  at  least 
equal  in  scope  to  the  cost  analysis  of  the  6883  test  station  and 
without  benefit  of  complete  production  and  installation  cost  data. 
Therefore,  the  effort  was  not  undertaken. 

These  problems  highlight  difficulties  associated  with 
attempting  a  cost  analysis  of  subsystem  components  in  the  absence  of  a 
total  system  analysis.  Although  the  6883  AET  and  the  simulators  are 
discrete  elements  in  the  training  system,  it  is  not  known  how  represen¬ 
tative  they  are  of  other  system  components.  To  illustrate  this  point, 
consider  that  the  cost  of  maintenance  for  the  6883  AET  was  based  on 
maintenance  records.  For  purposes  of  comparison  with  the  simulators, 
there  is  no  way  of  judging  whether  these  costs  are  representative  of 
the  other  test  stations.  Similar  problems  of  representativeness  arise 
with  respect  to  the  costs  of  courseware  development. 

To  summarize,  the  cost  analysis  should  be  understood  as  an 
effort  to  document  the  cost  experiences  for  alternative  training 
devices.  Since  available  cost  data  lack  the  detail  needed  for  general¬ 
izations  about  the  cost  of  simulation  at  the  training  system  level,  a 
fixed  effects  analysis  of  specific  training  devices  is  provided  in 
this  report. 


Student  Flow 


Another  important  aspect  of  the  evaluation  environment  is  the 
number  of  students  passing  through  the  training  program  in  a  specified 
time  period.  Table  1  presents  the  numbers  of  students  per  class  that 
participated  in  this  study  and  the  training/ testing  modes.  It  was  not 
uncommon  to  have  a  lapse  of  a  few  weeks  between  classes.  The  flow  of 
student  subjects,  while  anticipated  to  be  heavy,  was  characteristically 
low.  As  a  result,  though  very  few  students  were  lost  to  the  study,  it 
was  possible  to  evaluate  the  performances  of  only  129  students  during 
the  period  August  1980  through  October  1 98 1 . 


11 


Class  # 


Shift 


#  Students 


Training  Mode 


Testing  Mode 


NOTES:  Numbers  in  parentheses  indicate  class  size  prior  to  attrition  or 
washbacks. 

•Denotes  classes  used  for  pilot  testing. 

+Class  received  training  as  part  of  2-D  shakedown  procedure  and 
was  required  to  provide  questionnaire  data  only. 

++Class  lost  to  study  due  to  equipment  failure  in  mid-training. 
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Assignment  and  OJT  In  the  Field 


Once  trainees  are  assigned  to  the  field,  they  enter  OJT.  Field 
training  is  relatively  informal  and  is  designed  to  develop  the  skills 
needed  to  operate  and  maintain  the  assigned  test  station. 

Field  site  visits  conducted  during  the  course  of  the  project 
revealed  that  the  number  of  technicians  available  and  their  technical 
competency  upon  arrival  determined  the  extent  and  type  of  OJT  received. 
Due  to  the  individualized  approach  to  providing  training  in  the  field, 
it  was  not  possible  to  use  the  extent  and  type  of  OJT  required  as  an 
indicator  of  training  effectiveness.  A  second  method  of  assessing 
field  performance  was  considered.  This  approach  involved  recording 
the  number  and  cost  of  replacement  parts  requested  and  used  by  new 
technicians  to  perform  test  station  and  LRU  repairs.  However,  the 
implementation  of  this  indirect  measure  of  training  effectiveness  was 
not  feasible  due  to  the  complexities  of  obtaining  such  data  in  the 
field.  In  short,  it  was  difficult  to  isolate  clear  measures  of  long- 
range  impact  of  simulator  training. 

It  was  anticipated  that  the  performance  of  simulator- trained 
personnel  and  AET  trained  personnel  would  be  compared  after  specified 
intervals  of  time  in  the  field.  The  effects  of  OJT  were  expected  to 
be  present  and  constant  in  both  groups;  therefore,  any  differences  in 
performance  could  be  attributed  to  the  mode  of  training.  Clearly, 
with  variable  amounts  of  OJT,  this  assumption  was  not  valid.  In  fact, 
if  the  proposed  time  series  sampling  framework  was  used,  differences 
in  performance  due  to  training  would  be  reduced  as  time  in  the  field 
increased.  Therefore,  in  order  to  obtain  some  measure  of  the  long- 
range  impact  of  simulator  training  on  performance,  it  was  necessary  to 
devise  a  method  of  estimating  job  proficiency  prior  to  field  assignment, 
and  to  collect  subjective  ratings  of  field  performance  from  supervisors 
shortly  after  the  field  placement  (within  about  4  weeks). 

From  discussions  with  the  ATC  staff  and  field  personnel,  it 
became  apparent  that  field  assignments  are  made  in  view  of  the 
immediate  demand  for  specific  automatic  test  station  operators.  Thus, 
at  best  only  a  small,  undetermined  number  of  airmen  trained  to  operate 
the  6883  test  station  was  expected  to  be  assigned  to  the  6883  or  to 
the  highly  similar  6873  test  stations  in  the  field.  Although  the 
nature  of  the  field  assignments  remained  a  design  variable,  it  was 
clear  from  the  outset  that  extremely  disparate  sample  sizes  for 
students  assigned  to  6883  and  other  test  stations  would  be  obtained. 

In  fact,  as  discussed  more  fully  in  Chapter  V,  only  10  students  out  of 
129  were  assigned  to  the  6883  or  6873  test  stations  at  any  time  during 
the  first  4  weeks  of  assignment  to  duty  stations. 
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IV.  METHODOLOGY 


This  chapter  provides  a  discussion  of  all  data  collection  in¬ 
struments  developed  to  assess  both  classroom  and  field  performance  as 
well  as  costs.  The  techniques  used  for  data  collection  and  data 
management  are  also  outlined. 


Assignment  to  Treatment  Groups 


A  total  sample  of  129  F-111  avionics  maintenance  trainees  par¬ 
ticipated  in  this  study.  Twelve  additional  students  served  as  pilot 
subjects  during  revision  of  the  performance  measures  and  one  class  of 
six  students  was  eliminated  from  the  evaluation  because  training 
included  the  use  of  both  the  2-D  and  3-D  simulators.  A  complete 
schedule  of  6883  classes  participating  in  this  effort  was  provided  in 
Chapter  III  (rf.  Table  1). 

The  modified  experimental  design,  as  discussed  in  Chapter  II, 
included  five  cells  within  the  training  mode  by  testing  mode  matrix  to 
which  students  were  to  be  assigned.  However,  due  to  constraints 
imposed  on  the  implementation  of  that  design,  assignments  to  treatment 
groups  consistent  with  that  design  were  not  always  possible.  Rather 
than  eliminate  students  from  the  experiment,  complete  preassessment, 
performance,  and  follow-up  data  were  collected  regardless  of  eventual 
modes  of  training  and  testing.  The  resulting  distribution  of  students 
is  shown  in  Table  2.  Note  that  sample  sizes  approach  or  exceed  the 
expected  15  to  20  students  for  only  four  of  the  five  groups  proposed 
for  the  modified  design. 

While  it  was  intended  that  classes  be  randomly  assigned  to 
treatment  groups,  actual  assignments  were  primarily  determined  by 
equipment  availability.  Specifically,  Figure  9  shows  that  the  avail¬ 
ability  of  the  actual  equipment  was  generally  limited  to  the  early 
stages  of  the  data  collection  period,  whereas,  the  use  of  the  2-D 
simulator  was  restricted  to  later  stages  since  it  was  subject  to  a 
formative  evaluation  until  February  25,  1981.  This  inability  to 
provide  random  assignment  raised  a  number  of  questions  concerning 
possible  bias  in  treatment  groups.  Although  these  issues  are  fully 
considered  in  the  analysis,  previous  work  suggested  that  factors  such 
as  sex,  aptitude,  and  prior  ATC  achievement  do  not  pose  significant 
bias  problems  for  the  evaluation  of  6883  performance  (Cicchinelli  et 
al.,  1980).  However,  control  measures,  such  as  the  pre-assessment 
measures,  were  employed  to  measure  the  impacts  of  any  nonrandom 
assignment  upon  performance  criteria. 
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Figure  9.  Equipment  availability  for  training/testing  during  the  data  collection  period. 

Key:  —  Operational 

I  Not  available  due  to  malfunction 
l—  I  Not  available  due  to  reason  shown 


Table  2 


Distribution  of  Students  Across  6883  Training  and  Testing  Modes 


Training  Mode 

Testing  Mode 

AET 

3-D 

2-D 

Overall 

AET 

35» 

- 

- 

35 

3-D  Simulator 

14* 

26»+ 

8++ 

48 

2-D  Simulator 

4» 

10 

32* 

46 

All  Devices 

53 

36 

40 

129 

•Denotes  cells  consistent  with  modified  experimental  design. 
+Includes  five  students  with  patch  panel  training  on  the  Part-Task 
Trainer  rather  than  on  AET. 

++A11  patch  panel  training  accomplished  on  the  Part  Task  Trainer. 


In  addition  to  explaining  the  need  for  nonrandom  assignment  of 
subjects,  the  data  shown  in  Figure  9  also  graphically  illustrate  why 
it  was  not  possible  to  assign  a  sufficient  number  of  subjects  to  the 
2-D/AET  experimental  groups.  Further,  during  the  limited  period 
between  February  25  and  May  7,  1981  ,  when  both  the  2-D  and  AET  equip¬ 
ment  were  simultaneously  available,  student  flow  was  extremely  low 
(e.g.,  no  classes  in  March). 


Data  Collection 


Consistent  with  the  components  of  the  evaluation  design 
discussed  earlier  in  this  report,  four  general  types  of  information 
were  collected:  (a)  student  background  data,  (b)  classroom-related 
data,  (c)  field-related  data,  and  (d)  cost-related  data.  Procedures 
for  data  collection  were  carefully  designed  to  minimize,  as  much  as 
possible,  disruptions  of  normal  Air  Force  routines. 
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Student  Background  Data 


A  variety  of  aptitude  and  achievement  measures  were  collected 
to  examine  the  equivalence  of  treatment  groups.  Student  records 
provided  the  majority  of  this  background  information  including  ASVAB 
and  Air  Force  Qualifications  Test  (AFQT)  results  as  well  as  block 
scores  from  prior  ATC  training. 

Previous  research  by  Cicchinelli  et  al.  (1980)  found  no 
substantial  or  reliable  relationships  between  standard  Air  Force 
aptitude  measures  (ASVAB  and  AFQT)  and  student  performance  in  the  6883 
instructional  block.  Because  ASVAB  and  AFQT  scores  are  used  in  part 
to  determine  career  training  assignments,  such  tests  cannot  also  be 
expected  to  be  sensitive  to  differences  within  a  particular  course  of 
training.  For  the  present  evaluation,  therefore,  three  additional 
aptitude  tests  were  selected  for  administration  to  6883  students:  the 
Delta  Concealed  Figures  Test,  the  Delta  Reading  Vocabulary  Test, 2  and 
Form  A  of  the  Ship  Destination  Test. 2  Preassessment  was  conducted  in 
a  group  setting  at  the  beginning  of  each  6883  block  and  required 
approximately  45  minutes  to  complete. 


Classroom-Related  Data 

As  a  result  of  a  cooperative  agreement  between  ATC  and  AFHRL, 
one  day  at  the  end  of  the  Converter/Flight  Control  block  of  instruc¬ 
tion  was  made  available  to  DRI  for  data  collection.  This  temporary 
departure  from  the  normal  training  schedule,  effective  for  the 
duration  of  this  project,  allowed  data  to  be  collected  without 
altering  the  usual  6883  training  protocol.  Allocating  a  day  for 
testing  immediately  following  the  6883  training  was  extremely  helpful 
to  this  evaluati  <■  -  effort  for  a  number  of  reasons.  First,  the 
logistics  of  integrating  the  data  collection  into  ongoing  training 
were  simplified.  Second,  the  use  of  the  actual  or  simulated  6883  test 
stations  occurred  as  it  normally  would  if  there  had  been  no  evaluation 
effort.  Third,  the  availability  of  testing  time  immediately  following 
6883  training  eliminated  the  possibility  that  observed  performance 
would  be  affected  by  intervening  training.  Finally,  collecting 
performance  data  from  each  student  at  the  same  point  in  the  training 
sequence  ensured  that  all  students  nad  similar  levels  of  training  at 
the  time  of  testing.  Clearl.y,  the  cooperation  o'  'TC  in  this  matter 
was  essential  to  the  evaluation  effort  and  their  »  : istance  is  much 

appreciated . 


^These  tests  are  protected  by  copyright  laws.  Additional 
information  concerning  specific  test  items  can  be  obtained  from  the 
Air  Force  Human  Resources  Laboratory,  Lowry  Air  Force  Base,  Denver, 
Colorado. 


The  following  sections  provide  a  brief  discussion  of  the  major 
data  collection  instruments  developed  and  other  sources  of  classroom 
data. 


Hands-On  Troubleshooting  Test  (HOTT).  For  purposes  of  this 
evaluation,  maintenance  skills  were  largely  measured  in  terms  of 
electronic  troubleshooting.  This  emphasis  on  troubleshooting  perform¬ 
ance  testing  was  suggested  by  field  personnel  at  both  Plattsburgh  and 
Cannon  AFBs.  Twenty-four  avionics  personnel,  having  at  least  2  years 
of  field  experience,  were  asked  to  rate  the  relevance  of  the  6883- 
related  STS  standards  to  job  performance.  Troubleshooting  test  station 
malfunctions  was  considered  the  most  important  job  skill.  Respondents 
also  rated  the  ability  to  analyze  a  specific  problem  logically  as  a 
critical  skill.  Understanding  the  operation  of  the  test  station,  its 
component  TRUs,  and  associated  signal  flow  was  considered  necessary  to 
effectively  troubleshoot  the  station  and  LRUs.  The  mean  ratings  of 
all  STS  standards  by  field  personnel  were  presented  in  an  earlier 
report  (Cicchinelli  et  al.,  1980,  rf.  Chapter  IV,  Table  2). 

The  HOTT  was  designed  so  that  it  could  be  administered  on  the 
AET  or  either  the  2-D  or  3-D  simulator.  The  primary  focus  of  this 
test  was  on  troubleshooting  skills;  the  problem  selected  involved  the 
identification  and  correction  of  two  related  malfunctions,  one  in  an 
LRU  and  another  in  the  test  station.  The  test  was  designed  to  examine 
the  ability  of  a  student  to  perceive  that  test  station  and  its 
associated  LRUs  as  an  integrated  system.  The  two-step  nature  of  the 
malfunction  was  chosen  to  examine  the  procedural  and  logical  processes 
of  troubleshooting  under  circumstances  that  were  more  representative 
of  actual  on-the-job  malfunctions  than  of  6883  training  exercises. 

Since  the  test  was  dissimilar  to  the  lessons  available  on  both  the  2-D 
and  3-D  simulators  as  well  as  to  the  faults  typically  encountered  on 
the  AET,  the  task  was  new  to  all  students. 

The  HOTT  was  administered  and  individually  scored  by  DRI 
personnel. 3  Each  test  item  constitutes  a  discrete  step  that  a  student 
had  to  complete  in  order  to  identify  the  malfunction.  In  addition  to 
recording  the  completion  (or  noncompletion)  of  the  required  steps,  the 
nature  of  all  errors  committed  was  noted  on  the  scoring  form.  An 
analysis  of  these  errors  was  conducted  to  isolate  any  differences  among 
students  trained  on  the  6883  2-D  simulator,  3-D  simulator,  or  AET. 

Time  to  complete  various  portions  of  the  HOTT  was  also  recorded. 
However,  due  to  large  differences  in  machine  response  times,  vari¬ 
ability  in  the  need  for  student  prompting,  and  a  change  in  the  LRU 


3lnformation  concerning  this  performance  test  can  be  obtained 
from  the  Air  Force  Human  Resources  Laboratory,  Lowry  Air  Force  Base, 
Denver,  Colorado. 
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model  used  for  testing,  this  measure  was  subsequently  found  to  be 
inappropriate  for  purposes  of  comparing  student  performance.  The 
amount  of  information  provided  via  prompts  was  controlled  as  strictly 
as  possible  through  the  standardization  of  test  administration 
procedures.  However,  it  was  difficult  to  ensure  that  consistent 
information  via  prompts  was  always  provided.  Since  few  techricians 
use  the  same  logic  tree  to  solve  a  problem,  information  that  is  useful 
to  one  person  may  be  unenlightening  to  others  (Shriver  &  Foley,  1974). 


Paper-and-Pencil  Troubleshooting  Test  (PPTT).  Because  of  the 
importance  of  troubleshooting  skills,  a  paper-and-pencil  measure  was 
developed  in  cooperation  with  Air  Force  subject  matter  experts  to 
augment  the  hands-on  test.1*  The  PPTT  consisted  of  eight  discrete 
electronic  signal  flow  problems  of  one  or  more  parts  which  required 
students  to  use  their  TO  schematics  and  logical  skills  training. 


Since  this  test  was  developed  subsequent  to  the  initial  data 


collection  efforts,  it  was  available  for  administration  to  only  119  of 


the  129  students  in  the  study. 


Student  and  instructor  interviews.  All  students  participating 
in  the  study  were  asked  to  complete  a  brief  written  survey  concerning 
their  opinions  and  attitudes  about  the  use  of  simulators  in  training. 
The  survey  instrument  is  shown  in  Section  A-1  of  Appendix  A.  A 
written  format  was  selected  so  that  complete  and  consistent  daca  could 
be  collected  in  a  fashion  that  allowed  the  greatest  flexibility  within 
the  test  day  schedule. 


The  ATC  Instructor  Questionnaire  was  administered  in  October 
1981  to  nine  instructors  in  the  F-111  training  program  at  Lowry  AFB. 
Only  those  instructors  who  had  taught  the  6883  block  during  the  course 
of  the  evaluation  were  asked  to  respond.  Questions  regarding  the 
general  use  of  simulators  in  training  and  about  the  2-D  and  3-D 
simulators  in  particular  were  included.  This  questionnaire  is 
provided  in  Section  A-2  of  Appendix  A. 


^Information  concerning  this  performance  test  can  be  obtained 
from  the  Human  Resources  Laboratory,  Lowry  Air  Force  Base,  Denver, 
Colorado. 
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Course  content  and  equipment  malfunction.  To  determine  the 
nature  of  AET  use  in  training,  course-related  documents  were  collected 
and  reviewed,  instructors  and  students  were  interviewed,  and  direct 
observations  of  classroom  proceedings  were  conducted  prior  to  the 
performance  evaluation  phase.  Additionally,  the  specific  training  re¬ 
ceived  by  students  on  the  6883  AET  and  both  simulators  was  recorded. 

The  training  received  was  categorized  according  to  the  TRUs  of  the  AET 
or  simulators  addressed  in  the  lesson,  the  LRUs  which  were  included  in 
the  training  period,  and  the  complexity  of  the  training  as  defined  by 
ATC  instructors  and  software  developers. 

Equipment  malfunctions  for  all  three  trainers  were  also 
recorded.  This  was  accomplished  by  reviewing  maintenance  department 
records  for  the  AET  and  2-D  and  3-D  simulators.  In  addition,  appro¬ 
priate  instructors  were  interviewed  to  corroborate  the  written  records 
for  all  three  trainers.  Equipment  malfunction  data  were  collected  on 
all  malfunctions  which  in  any  way  interrupted  or  interfered  with  the 
training  of  the  avionics  technicians.  Since  all  trainers  are 
"scheduled"  to  be  available  for  training  each  day,  equipment  downtime 
(i.e.,  unavailability  for  classroom  use),  regardless  of  the  availability 
of  "substitute"  trainers,  was  logged.  This  information  served  a  dual 
purpose:  to  track  any  effects  of  malfunctions  on  training  and  to  pro¬ 
vide  information  for  cost  analysis. 

Data  were  also  collected  regarding  training  deviations  for 
students  during  the  period  of  their  6883  training.  Reasons  for 
deviations  range  from  adverse  weather  conditions  to  events  authorized 
by  the  Base  Commander  to  equipment  breakdowns.  Training  deviations 
are  given  to  students  who  have  not  received  all  of  the  required 
training  time  in  a  particular  block. 


Predicting  Job  Proficiency 

One  initial  objective  of  this  research  was  to  assess  the 
possible  impact  of  simulator  training  on  actual  job  performance.  As 
discussed  in  earlier  reports  (Cicchinelli  et  al,  1980),  differences  in 
job  performance  as  a  function  of  training  equipment  would  be  obscured 
by  the  variable  nature  of  OJT.  To  circumvent  this  problem,  the  aid  of 
Air  Force  field  supervisors  was  enlisted  to  develop  a  paper-and-pencil 
test  to  serve  as  a  predictor  of  future  job  performance.  This  instru¬ 
ment  was  administered  on  the  test  day  immediately  following  6883 
training. 


To  insure  the  validity  of  the  Projected  Job  Proficiency  Test 
(PJPT),  criteria  of  job  proficiency  were  established.  Questions  were 
solicited  from  field  supervisors  which  would  reflect  levels  of  knowledge 
expected  of  newly  trained  personnel.  A  total  of  75  items  were  received. 


40 


From  these  items  and  questions  obtained  from  Lowry  AFB  instructors,  an 
initial  70-item  pencil-and-paper  test  of  job  proficiency  was  developed. 
An  item  analysis  previously  conducted  on  115  students  in  earlier 
research  (Phase  II)  resulted  in  revisions  to  the  instrument.  The  final 
form  of  the  PJPT  included  35  items  and  was  more  objective  and  reliable 
than  the  original  version. 5  This  instrument  required  only  50  minutes 
to  complete. 


Field-Related  Data 

Subsequent  to  permanent  field  assignment,  all  technicians  who 
participated  in  this  evaluation  project  and  their  field  supervisors 
completed  follow  up  questionnaires.  Since  former  students  could  be 
assigned  to  any  one  of  three  work  shifts  and  were  sometimes  on 
temporary  leave  for  medical  or  educational  reasons,  locating  them  was 
itself  a  difficult  task.  Supervisory  field  personnel  were  instrumental 
in  this  aspect  of  the  data  collection  effort.  They  assumed  the 
responsibility  of  distributing  the  questionnaires  and  forwarding  all 
information  to  DRI.  Generally,  the  technicians  and  their  supervisors 
completed  the  questionnaires  at  their  assigned  stations. 

Approximately  15  minutes  of  each  person's  time  was  required. 

Field  technician  survey.  By  means  of  an  interview  question¬ 
naire,  technicians  who  had  participated  in  the  evaluation  program  and 
were  subsequently  assigned  to  the  field  were  asked  to  rate  the 
adequacy  of  their  ATC  F-111  training  and  its  relationship  to  their 
field  work.  Those  technicians  who  had  contact  with  either  simulator 
during  training  were  asked  specifically  to  rate  the  adequacy  of 
simulator  training.  Section  A-3  of  Appendix  A  includes  this 
questionnaire. 


Field  supervisor  survey.  Technicians  in  OJT  are  assigned 
supervisors  who  assist  them  in  becoming  familiar  with  the  responsi¬ 
bilities  of  field  assignments  and  who  rate  their  performances  during 
the  OJT  period.  The  supervisors  may  work  at  the  assigned  test  station 
with  the  technicians  or  may  only  be  available  to  answer  questions  on 
an  as-needed  basis.  Personnel  limitations  require  that  one  supervisor 
be  responsible  for  more  than  one  technician  in  most  cases.  These 
supervisors  were  asked  to  rate  the  performances  of  technicians  in 
areas  of  housekeeping,  use  of  testing  equipment,  knowledge  of  LRU  and 
TRU  circuit  flow  and  operation,  and  troubleshooting  initiative.  There 
was  a  total  of  23  items  on  which  each  technician  wa  'ited. 


^Information  concerning  this  performance  test  can  be  obtained 
from  the  Human  Resources  Laboratory,  Lowry  Air  Force  Base,  Denver, 
Colorado. 
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Field  follow  up  data  were  collected  once  a  technician  had  been 
involved  with  OJT  for  approximately  1  month.  Based  on  previous  work, 
it  was  decided  that  training  effects  would  be  further  diluted  by  prac¬ 
tical  field  experience  if  a  longer  lag  time  was  selected.  The  super¬ 
visor  rating  form  is  included  as  Section  A-4  of  Appendix  A. 


Cost-Related  Data 


A  comparison  of  6883  2-D  simulator,  3-D  simulator,  and  AET 
life  cycle  costs,  under  the  assumption  of  equal  training  effectiveness 
was  conducted  to  determine  which  trainer  was  most  cost  effective. 

Since  the  6883  simulators  are  among  the  first  such  devices  introduced 
into  the  intermediate  level  maintenance  training  environment,  this 
analysis  also  served  as  a  point  of  reference  for  a  discussion  of 
simulation  options  available  in  Air  Force  maintenance  training. 


The  question  of  which  trainer  is  most  cost  effective  can  be 
complicated  when  different  training  equipment  utilization  patterns  for 
the  simulators  and  AET  produce  differences  in  training  effectiveness. 
For  example,  the  most  costly  equipment  alternative  might  still  be  the 
most  .post  effective  because  the  equipment,  together  with  its  utiliza¬ 
tion  strategy,  produces  proportionately  greater  training  effectiveness 


These  complications  were  not  encountered  in  the  evaluation  be¬ 
cause  the  total  training  environment  was  not  disturbed  by  the  introduc 
tion  of  the  simulators.  That  is,  with  the  introduction  of  the  simula¬ 
tors,  no  new  training  strategy  was  imposed  which  could  have  been 
expected  to  result  in  differences  in  training  effectiveness.  Thus, 
the  comparison  of  only  the  life  cycle  costs  of  the  three  trainers  is  a 
realistic  way  to  establish  which  is  most  cost  effective. ^ 


^It  should  be  noted  that  the  term  "cost  effectiveness"  is  used 
here  in  a  limited  sense:  which  of  two  or  more  equivalent  training 
systems  has  the  least  total  cost  of  ownership.  The  notion  of  cost  ef¬ 
fectiveness  of  simulation  in  training  is  a  broad  concept  that  presumes 
(a)  equal  effectiveness  of  both  simulators  and  AETs  and  (b)  that 
student  performance  can  be  gauged  by  objective  performance  criteria 
associated  with  student  task  performance  on  the  actual  equipment.  In 
this  context,  cost  effectiveness  analysis  of  simulation  training  would 
include  a  determination  of  the  marginal  utility  of  the  simulators, 
that  is,  at  what  point  does  greater  use  of  a  simulator  no  longer 
reduce  training  costs  for  the  simulator-AET  combination?  Orlansky  and 
String  (1977)  develop  this  important  topic  in  their  review  of  Cost- 
Effectiveness  of  Flight  Simulators  for  Military  Training.  This  issue 
is  not  engaged  in  the  present  cost  analysis  because  the  amount  of 
training  could  not  be  systematically  varied  (for  institutional 
reasons),  nor  could  student  performance  of  troubleshooting  tasks  be 
ascertained  using  existing  Air  Force  performance  criteria.  This 
assessment  is  not  intended  to  be  a  criticism;  rather  it  is  a  comment 
on  the  maturity  of  simulation  development  for  the  maintenance  training 
field. 
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Life  cycle  cost  comparison.  The  approach  taken  to  the 
development  of  the  life  cycle  cost  comparison  considers  that  each 
simulator  is  a  replacement  alternative  to  the  AET.  The  respective 
cost  streams  for  each  of  the  trainers  were  calculated  for  a  projected 
15-year  life.  Since  the  comparison  used  1978  as  the  reference  year, 
AET  costs  incurred  prior  to  1978  and  simulator  costs  incurred 
subsequent  to  1978  were  adjusted  according  to  the  annual  inflation 
rate  to  establish  investment  costs  in  1978  dollars.  Operating  costs 
were  discounted  at  10  percent  annually  to  establish  net  present  value 
(NPV)  in  1978  dollars.' 

Figure  10  displays  the  framework  for  cost  comparison  and  the 
major  categories  for  which  input  data  were  acquired.  The  Facilities 
category  refers  to  the  cost  of  space  necessary  to  house  the  training 
device.  The  Equipment  category  is  composed  of  four  subcategories:  (a) 
Cost  of  Hardware,  including  original  software,  (b)  Specifications  and 


Cost  Categories  AET  2-D  Simulator  3-D  Simulator 


Facilities 

Equipment 

Instructional  Materials 

Personnel 

Students 

Miscellaneous 


TOTAL  (construction  $) 


NPV  (1978) 


Cost  Effectiveness 
($  per  student-hour) 


Figure  10.  Life  cycle  cost  comparison  framework. 


?This  discount  rate  is  consistent  with  Office  of  Management 
and  Budget  (0MB)  Guidelines  (Circular  A94). 


Acquisitions  Management,  which  includes  personnel  time  for  developing 
contract  specifications,  contract  award  and  monitoring,  (c)  Support 
and  Installation,  which  includes  personnel  costs  incurred  after 
delivery  of  the  device  for  set-up,  debugging,  and  formative  evaluation 
(excluding  on-site  manufacturer  costs),  and  (d)  Sustaining  Investment, 
which  includes  the  costs  of  maintenance  (spares  and  personnel  time  for 
repair).  The  Instructional  Materials  category  includes  the  cost  of 
lessonware  revision  and  the  development  of  additional  software/ 
lessonware  needed.  The  Personnel  category  includes  the  costs  incurred 
for  administration  and  instruction,  and  the  Student  category  refers  to 
student  wages  and  support.  Finally,  the  Miscellaneous  category  was 
included  to  account  for  supplies  regularly  used  in  laboratory  training. 

Related  cost  models.  Two  studies  concerned  with  the  cost  of 
training  systems  have  been  particularly  useful  in  this  work.  The  Naval 
Training  Analysis  and  Evaluation  Group  (TAEG)  developed  a  cost  model 
(Braby,  Henry,  Parrish,  &  Swope,  1978)  which  included  methods  for 
evaluating  the  elements  in  each  cost  category  sho.,n  above.  A  computer 
program  for  calculating  total  training  cost  was  also  provided.  The 
TAEG  model  and  its  cost  element  estimation  procedures  would  have  been 
used  in  this  work  except  for  two  drawbacks:  (a)  the  model  contains 
insufficient  detail  concerning  the  equipment  acquisition  life  cycle 
phase  that  was  considered  important  here,  and  (b)  the  emphasis  given 
the  TAEG  model  was  one  of  predicting  costs  of  alternative  systems  for 
the  purpose  of  cost  acquisition  minimization,  not  for  predicting  total 
costs  of  system  ownership. 

The  AFHRL  Logistics  Research  and  Training  Division,  Wright- 
Patterson  AFB,  completed  a  life  cycle  cost  estimation  for  F-16 
simulated  and  AET  maintenance  training  systems  in  which  the  Braby  model 
was  used  as  a  starting  point  (Eggemeier,  1979;  Eggemeier  &  Klein,  1979, 
1981).  In  addition,  a  logistics  support  cost  (LSC)  model  was  incorpo¬ 
rated  to  estimate  elements  in  the  equipment  cost  category.  The  latter 
model  was  originally  developed  for  the  Simulator  Systems  Program  Office 
(Sim  SPO)  to  estimate  costs  for  aircrew  simulators.  The  principal 
value  of  the  LSC  model  is  that  it  can  be  used  to  estimate  certain  cost 
elements  based  on  functional  relationships  derived  from  historical 
acquisition  data.  This  technique  was  helpful  in  the  present  study  for 
estimating  sustaining  investment  costs  associated  with  the  6883  AET 
operation.  In  addition,  the  F-16  analysis  also  provided  guidance  in 
estimating  elements  in  other  cost  categories. 

The  cost  model  presented  in  this  study  is  perhaps  best  thought 
of  as  a  method  for  organizing  and  tabulating  cost  elements  ("ingre¬ 
dients")  for  which  historical  data  are  available.  The  sources  of  the 
historical  data  for  the  AET  station  used  in  this  co3t  comparison 
included  the  3^50th  Technical  Training  Group,  Avionics  Branch  and  the 
Logistics  Support  Group  (Lowry  Are).  The  sources  of  data  for  the 
simulators  include  the  AFHRL  technical  monitors  for  the  simulator 
contracts  and  the  manufacturer  of  the  3-D  simulator. 
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After  students  completed  the  F-111  training  program,  training 
and  performance  data  were  collected,  coding  forms  were  completed  and 
the  data  were  entered  into  a  computer  file  for  subsequent  analysis. 

All  data  except  field  follow  up  data  for  the  last  31  students  were 
collected  for  each  student.  Due  to  the  approximately  12  week  time 
period  between  completion  of  the  6883  block  of  training  and  1  month  at 
the  field  assignment,  these  data  were  not  available  for  this  analysis. 

The  data  collected  regarding  training  deviations,  training  re¬ 
ceived,  unscheduled  maintenance,  and  personal  interviews  were  also  or¬ 
ganized  for  analysis.  When  the  evaluation  period  was  concluded,  the 
impact  of  these  various  factors  on  training  and  performance  was 
assessed. 
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V.  RESULTS  AND  DISCUSSION 


The  primary  focuses  of  this  cost  and  training  effectiveness 
evaluation  were:  (a)  to  examine  classroom  and  field  performance  as  a 
function  of  training  mode  (AET,  2-D  simulator,  or  3-D  simulator),  (b) 
to  compare  and  contrast  the  characteristic  operation  of  each  trainer, 
and  (c)  to  assess  the  costs  of  using  the  alternative  types  of  training 
equipment.  This  chapter  provides  a  discussion  of  all  evaluation 
findings  related  to  these  three  objectives  as  well  as  the  results  of 
all  supporting  analyses  undertaken.  Because  of  the  stringent  selection 
criteria  used  for  entry  into  avionics  technician  training,  students 
were  fairly  homogeneous  with  respect  to  both  inherent  abilities  and 
achievements  in  the  overall  ATC  program  of  instruction.  Furthermore, 
marked  differences  in  subsequent  performance  could  not  be  expected 
after  as  brief  a  training  manipulation  as  in  the  present  case.  Thus, 
any  tests  developed  to  isolate  training  differences  had  to  be  partic¬ 
ularly  sensitive  to  differences  in  the  training  equipment. 


Analysis  of  Troubleshooting  Performance 


A  central  issue  to  be  addressed  by  this  investigation  was 
whether  2-D  simulator,  3-D  simulator,  and  AET  training  would  result  in 
equal  levels  of  student  performance  on  troubleshooting  problems  as 
might  be  encountered  in  the  field.  As  discussed  in  Chapter  IV,  three 
separate  measures  of  6883  training  effectiveness  were  employed.  The 
primary  measure  was  a  timed,  serial,  26-item  practical  test  (HOTT)  of 
each  student's  ability  to  troubleshoot  malfunctions.  This  test  was 
supplemented  by  the  PPTT  and  Air  Force-administered  6883  block  test 
results. 


These  data  will  be  considered  in  the  sections  that  follow. 
Because  the  precision  of  performance  measurement  was  a  primary  concern, 
and  because  nonrandom  assignment  of  students  to  training  modes  may 
have  resulted  in  biased  experimental  groups,  an  analysis  of  covariance 
(ANCOVA)  framework  was  considered  most  appropriate  for  the  evaluation 
of  performance-related  data.  The  procedures  used  to  identify  suitable 
covariates  for  these  analyses  are  described  in  Appendix  B  along  with  a 
summary  of  all  pretraining  variables  considered  in  this  analysis. 


Hands-On  Troubleshooting  as  a  Function  of  Training 

Table  3  summarizes  student  performance  on  the  HOTT  as  a  func¬ 
tion  of  training  and  testing  modes.  While  all  129  students  may  have 
completed  a  particular  test  item,  only  126  overall  test  scores  are 


46 


shown.  This  is  due  to  the  method  of  scoring  the  test.  For  those 
students  who  were  able,  for  whatever  reasons,  to  isolate  the  cause  of 
the  malfunction  (the  blown  fuse)  without  completing  all  of  the  inter¬ 
mediate  steps,  no  overall  score  was  recorded.  Instead,  data  for  those 
items  completed  were  included  insofar  as  they  were  in  correct  sequen¬ 
tial  order. 

Two  hypotheses  were  of  particular  interest  with  respect  to 
hands-on  troubleshooting  abilities: 

1.  Students  trained  on  the  3-D  simulator,  the  2-D 
simulator,  and  the  6883  AET  are  equally  accurate 
in  solving  troubleshooting  problems. 

2.  Students  trained  on  the  3-D  simulator,  the  2-D 
simulator,  and  the  6883  AET  operate  the  AET  with 
equal  proficiency. 

Table  3 

Mean  Hands-On  Troubleshooting  Test  Scores 
by  Training  and  Testing  Modes 


Testing  Mode 


Training  Mode 

AET 

3-D 

2-D 

N 

34 

AET 

X 

21.50 

- 

- 

a 

2.16 

3-D  Simulator 

N 

X 

CT 

14 

22.57 

1.79 

26 

21.58 

2.19 

8 

21.00 

2.83 

N 

U 

10 

30 

2-D  Simulator 

X 

24.25 

21.60 

22.30 

a 

0.96 

1.811 

1.73 

appropriate  comparison  in  lieu  of  a  full  factorial  design  since  all 
students  were  equally  familiar  with  the  testing  equipment.  Further¬ 
more,  this  approach  avoided  making  the  assumptions  necessary  to 
collapse  across  testing  modes,  while  including  70  percent  of  the 
available  student  data.  Scores  from  the  Delta  Concealed  Figures  Test, 
the  General  Aptitude  Test,  and  the  Common  Automatic  Test  Equipment 
(CATE)  instructional  block  served  as  covariates  for  this  and 
subsequent  analyses  performed  on  the  HOTT  results.  The  analysis 
revealed  no  significant  differences  as  a  function  of  6883  training 
mode  (F(2,77)  =  1.95,  £  =  .15). 

The  second  hypothesis  required  a  comparison  of  the  AET  tested 
treatment  groups  as  a  function  of  training.  Unfortunately,  only  one 
class  of  four  students  could  be  tested  on  the  AET  following  2-D 
training;  thus,  a  direct  test  of  this  hypothesis  was  not  possible. 
However,  the  rationale  can  be  developed  that  the  2-D  and  3-D  sim¬ 
ulators  should  be  considered  as  "AET  equivalent"  for  purposes  of 
testing.  First,  previous  research  by  Cicchinelli  et  al.  (1980)  found 
no  effect  of  testing  mode  (3-D  simulator  vs.  AET)  on  the  accuracy  of 
student  troubleshooting  performance.  Second,  both  simulators  are 
fairly  high  fidelity  replicas  of  the  6883  test  station  and  thus  are 
reasonable  alternatives  to  the  AET.  Third,  the  present  data  showed  no 
difference  among  the  3-D  simulator-trained  students  as  a  function  of 
testing  mode  (F  (2,39)  =  1.05,  £  =  .36).  And  fourth,  an  ANCOVA 
comparison  of  troubleshooting  scores  for  2-D  simulator-trained 
students  who  were  tested  on  either  the  2-D  or  3-D  equipment  also 
showed  no  significant  main  effect  of  testing  (F  (1,35)  =  1.15,  £= 
.29).  The  2-D/AET  condition  was  not  included  as  a  treatment  level  in 
this  latter  analysis  because  so  few  observations  were  available.  In 
summary,  it  seemed  unlikely  that  troubleshooting  test  results  were  af¬ 
fected  by  the  device  used  for  test  administration.  For  these  reasons, 
it  was  assumed  that  all  testing  modes  might  be  combined  in  order  to 
examine  "AET  equivalent"  testing. 

Turning  once  again  to  hypothesis  #2,  AET,  2-D,  and  3-D  testing 
modes  were  combined,  and  data  from  the  resulting  three  training  groups 
were  analyzed  in  the  usual  ANCOVA  framework.  A  marginally 
significant  effect  of  training  mode  was  found  with  this  approach  (F 
(2,112)  =  2.79,  £  =  .07).  Subsequent  tests  of  orthogonal  contrasts 
showed  that  while  there  was  no  difference  in  performance  between  2-D 
and  3-D  trained  students  (F  (1,112)  <  1),  simulator  and  AET  trained 
students  did  differ  reliably  (F  (1,112)  =  4.95,  £  =  .03).  It  should 
be  stressed  that  two  important  and  somewhat  dubious  assumptions  have 
been  made  in  arriving  at  this  finding.  First,  it  was  assumed  that  the 
testing  mode  had  no  effect  on  2-D  trained  student  performance,  based 
only  on  data  from  two  of  the  three  testing  levels.  And  second,  it  was 
assumed  that  there  was  no  interaction  between  training  and  testing.  A 
violation  of  either  assumption  was  possible  and  could  account  for  this 
marginal  difference  in  performance. 
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Training/Testing  and  Specific  Troubleshooting  Test  Item3 


Using  Cronbach's  method,  the  reliability  of  the  HOTT  was 
computed  and  found  to  be  low  (a  =  .37),  suggesting  that  a  variety  of 
skills  were  contributing  to  each  total  score.  For  this  reason,  and 
because  meaningful  differences  in  overall  scores  had  not  been  shown  as 
a  function  of  training,  a  more  detailed  analysis  of  the  data  was 
performed.  Error  rates  for  each  of  the  26  items  were  computed  and  are 
presented  in  Table  4  according  to  student  training  modes. 


Table  4 

Mean  Error  Rates  for  Hands-On  Troubleshooting  Test 
Items  as  a  Function  of  6883  Training  Modes 


Item  it 

Training  Mode 

AET 

3-D 

2-D 

1 

.09 

.15 

.22 

2 

.03 

.15 

.04 

3 

.03 

.06 

.00 

4 

.03 

.06 

.00 

5 

.14 

.13 

.15 

6 

.00 

.04 

.00 

7 

.06 

.02 

.00 

8 

.03 

.02 

.00 

9 

.09 

.00 

.00 

10 

.03 

.10 

.04 

11 

.09 

.06 

.11 

12 

.03 

.08 

.13 

13 

.03 

.08 

.07 

14 

.17 

.10 

.15 

15 

.03 

.04 

.09 

16 

.03 

.06 

.00 

17 

.23 

.15 

.04 

18 

.09 

.13 

.13 

19 

.59 

.40 

.42 

20 

.41 

.56 

.36 

21 

.18 

.23 

.05 

22 

.18 

.33 

.18 

23 

.62 

.58 

.43 

24 

.76 

.50 

.75 

25 

.12 

.04 

.09 

26 

.24 

.27 

.16 
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A  review  of  the  data  shown  in  Table  4  suggests  the  following  observa¬ 
tions: 

•  Safety  errors  (item  4*1)  were  slightly  more  common 
among  simulator-trained  students  than  among  those 
trained  on  the  AET.  Furthermore,  while  8l  percent 
of  the  simulator-trained  students  were  assigned  to  a 
simulator  for  hands-on  testing,  they  accounted  for 
94  percent  of  the  errors  made  by  simulator-trained 
students.  Thus,  it  appears  that  an  error  on  item  4*1 
is  more  likely  with  simulator  training  and  testing. 

This  finding  suggests  that  students  may  have  failed 
to  perceive  the  simulators  as  actual  test  stations. 

Special  efforts  should  be  directed  toward  ensuring 
that  safety  is  not  compromised  by  this  apparent 
difference  in  student  attitudes. 

•  Students  had  particular  difficulty  with  items  4*19 
and  4*20 — recognizing  the  need  to  rerun  a  test  that 
has  failed  by  returning  to  the  previous  "0-ending" 
test  number.  It  was  expected,  based  on  DRI's  class 
observations,  that  AET  trained  students  would  be 
more  likely  to  recognize  the  need  to  repeat  a  test 
that  has  failed  (item  4*19)  but  less  likely  to  choose 
the  correct  method  (item  4*20).  This  followed  from 
the  observations  that  AET  test  failures  are  quite 
common,  and  they  are  generally  addressed  by  selecting 
the  "repeat"  mode  option  which  was  not  a  correct 
procedure  for  the  hands-on  test.  Both  groups  of 
simulator-trained  students  were  expected  to  make 
more  errors  on  the  "need  to  repeat  test"  item,  since 
this  step  is  not  stressed  by  the  training  lessons. 
Surprisingly,  the  error  rate  results  were  at  odds 
with  both  of  these  predictions.  AET  trained  students 
were  somewhat  less  likely  to  recognize  the  need  to 
repeat  the  test  that  failed  and  3-D-trained  students 
were  least  likely  to  select  the  proper  repeat  pro¬ 
cedure. 

•  Item  4*21  dealt  with  suggesting  replacement  of  the 
faulty  component  (Yaw  Board  TB3).  The  fact  that  2-D 
students  performed  slightly  better  than  3-D  or  AET 
groups  on  this  item  may  be  due  to  the  nature  of  the 
student  action  panel  used  for  2-D  student-simulator 
interactions.  The  single  "Repair  or  Replace"  option 
is  a  common  solution  to  2-D  lessons. 
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•  In  general,  as  items  #23  and  #24  reflect,  decoding 
the  fault  isolation  proved  to  be  a  difficult  task 
for  all  students.  While  the  simulator  training 
modes,  and  3-D  in  particular,  showed  a  slight  edge 
in  performance  on  these  items,  the  difference  is  not 
likely  to  be  reliable. 

Overall,  the  differences  observed  in  item  error  rates  as  a 
function  of  training  mode  were  minimal.  This  finding  is  consistent 
with  the  analyses  of  total  scores  discussed  previously. 


Supplemental  6883  Performance  Measures 

Two  additional  measures  of  6883  performance  were  recorded — the 
Air  Force-administered  6883  end-of-block  test  and  the  PPTT.  Table  5 
presents  the  mean  scores  for  each  as  a  function  of  the  three  training 
modes.  Following  the  statistical  approach  outlined  in  Appendix  B, 
covariates  were  selected  for  each  measure.  For  6883  block  scores, 

Table  5 

Supplemental  6883  Performance  Measures 
as  a  Function  of  Training  Mode 


Training 

Mode 

Measures 

6883  Block  Score 

PPTT  Score 

N 

35 

31 

AET 

X 

85.26 

19.29 

a 

8.70 

6.65 

3-D 

N 

48 

42 

Simulator 

X 

84.00 

18.67 

0 

9.64 

7.90 

2-D 

N 

46 

46 

Simulator 

X 

83.39 

19.78 

a 

10.57 

7.52 
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these  covariates  consisted  of  Administrative  Aptitude  Scores  and 
scores  from  the  Electronic  Fundamentals  and  CENPAC  instructional 
blocks.  For  the  PPTT,  scores  from  the  Electronic  Fundamentals  block 
as  well  as  Electronic  and  Mechanical  Aptitude  Test  results  were 
selected  as  covariates.  Separate  ANCOVAs  revealed  no  significant 
differences  among  training  conditions  for  either  the  6883  block  scores 
(F  (2,111)  <  1 )  or  the  written  troubleshooting  scores  (F  (2,102)  = 
1.12,  £  =  .33). 

In  sum,  students  trained  on  the  simulators  and  those  trained 
on  the  AET  did  not  differ  appreciably  with  respect  to  overall  trouble¬ 
shooting  abilities  as  measured  by  both  practical  and  written  tests. 
Furthermore,  when  students  were  compared  across  an  "AET  equivalent" 
testing  mode,  all  three  training  modes  exhibited  similar  levels  of 
troubleshooting  proficiency. 


Estimation  of  Field  Preparedness 


Two  methods  were  used  to  assess  the  impact  of  training  mode  on 
student  acquisition  of  field-relevant  skills.  One  analysis  dealt  with 
the  results  of  the  PJPT  which  was  administered  to  all  students  at  the 
conclusion  of  the  6883  instructional  block.  As  discussed  previously 
in  this  report,  the  PJPT  was  specifically  developed  for  this  purpose 
(rf.  Chapter  IV).  Since  subsequent  instructional  blocks  included 
training  components  similar  to  those  in  OJT,  student  performance  in 
those  blocks  also  indicated  field  preparedness.  Therefore,  the  second 
analysis  examined  the  results  of  Air  Force-administered  achievement 
tests  for  these  later  blocks. 


Projected  Job  Proficiency  as  a  Function  of  Training  Mode 

The  PJPT  was  a  paper-and-pencil  test  that  included  35  four- 
alternative  choice  items,  each  contributing  equally  to  a  total  score. 
The  mean  PJPT  scores  were  20.8,  21.0,  and  20.6  for  AET,  3-D,  and  2-D 
trained  students,  respectively.  Not  surprisingly,  these  values  did 
not  differ  appreciably  when  subjected  to  an  ANCOVA  (F  (2,119)  <  1 )  in 
which  scores  from  the  Electronic  Fundamentals  and  DATAC  blocks  and  the 
Ship  Destination  Test  served  as  covariates.  Thus,  the  hypothesis  was 
supported  that  the  3-D  simulator,  the  2-D  simulator,  and  the  AET  pro¬ 
vide  equivalent  job-related  experiences. 


Subsequent  Classroom  Performance  as  a  Function  of  6883  Training 

Another  facet  of  assessing  possible  training  effects  on  job- 
related  abilities  was  the  analysis  of  subsequent  instructional  block 
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scores.  It  was  anticipated  that  practical  blocks  would  be  particularly 
sensitive  to  simulator  versus  actual  training  experiences  in  the  6883 
block  and  also  most  similar  to  the  eventual  field  training  situation. 
Four  of  the  nine  subsequent  training  blocks  involved  hands-on  training, 
but  no  training  differences  could  be  found  since  all  students  received 
a  grade  of  "satisfactory"  in  each. 

Since  practical  instruction  blocks  offered  no  useful  data, 
performance  in  subsequent  theory  instruction  blocks  was  examined  as  an 
alternate  means  of  assessment.  Table  6  presents  the  mean  block  scores 
for  the  five  theoretical  courses  as  a  function  of  6883  training  mode. 

A  series  of  univariate  analyses  of  covariance  were  performed  in  order 
to  assess  the  possible  effects  of  6883  training  on  subsequent  block 
scores.  Covariates  were  again  selected  according  to  the  procedures 
outlined  in  Appendix  B,  and  varied  somewhat  for  each  measure  as  shown 
in  Table  6.  Only  in  the  case  of  the  Attitude  and  Rate  TS  block  was 
there  even  a  marginally  significant  effect  of  training  on  performance. 
However,  while  a  minor  difference  was  detected,  the  overall  data  sug¬ 
gest  that  students  perform  equally  well  on  subsequent  instructional 
blocks  whether  trained  on  the  3-D  simulator,  the  2-D  simulator,  or  the 
AET. 


User  Acceptance 


User  acceptance  of  the  3-D  and  2-D  simulators  was  assessed 
through  the  administration  of  student  interviews  and  instructor  inter¬ 
views.  Students  were  given  interview  forms  at  the  completion  of  DRI 
end-of-block  testing;  thus  student  acceptance  was  based  on  responses 
of  all  129  students  who  participated  in  this  evaluation  effort. 
Instructor  acceptance  was  examined  on  the  basis  of  interview  forms 
given  to  nine  instructors  who  were  involved  in  delivering  6883  training 
during  the  project  period.  The  amount  of  contact  time  with  either  or 
both  of  the  simulators  varies  considerably  among  instructors.  This 
was  primarily  due  to  the  existence  of  a  Special  Projects  Group  com¬ 
posed  of  three  F-111  instructors  who  were  responsible  for  conducting 
all  6863  simulator-based  training.  AET  instruction  was  provided  by 
several  additional  instructors  who  were  given  fewer  opportunities  to 
operate  or  observe  the  simulators. 


Student  Interviews 


verall,  students  did  not  differ  greatly  in  their  impressions 
of  the  6883  instructional  block  as  a  function  of  training  mode.  Inter¬ 
view  items  were  rated  on  a  5-point  scale,  where  5  corresponded  to  a 
highly  positive  response.  The  mean  ratings,  shown  in  parentheses, 


Table  6 


Instructional  Block  Performance 
Subsequent  to  6883  Training 


686J  Training  Mode 


Instructional  Block 

AET 

3-D 

Simulator 

2-D 

Simulator 

ANCOVA* 

F-Ratio 

Computer  Test 

N 

27 

29 

23 

Station  (TS) 

X 

92.8 

90.5 

89.6 

1.56 

a 

6.3 

6.9 

6.2 

(£=.22) 

Attitude  and 
Rate  TS 

N 

X 

0 

27 

85.1 

9.4 

29 

84.0 

8.7 

23 

89.4 

6.5 

2.42 

(£=.10) 

N 

8 

19 

23 

Display  TS 

X 

87.3 

88.0 

84.5 

<  1 

0 

12.5 

8.8 

8.8 

Video  TS 

N 

X 

o 

35 

80.7 

11.0 

48 

82.4 

9.0 

46 

79.5 

13.5 

1.13 

(£=.33) 

Radar-Transmitter 

N 

35 

48 

41 

Modulator  TS 

X 

89.5 

87.8 

90.5 

<  1 

o 

6.2 

9.3 

7.7 

•Selected  covariates  were  as  follows: 

Computer  TS — Sex,  Electronic  Fundamentals,  Navigation  Block  Scores 
Attitude  and  Rate  TS — CENPAC  Block  and  Ships  Destination  Scores 
Display  TS — Electronic  Fundamentals,  Logic,  and  AGE  Principles  Block 
Scores 

Video  TS — Navigation  and  Electronic  Systems  Block  Scores 
Radar-Transmitter  Modulator  TS — Concealed  Figures  Test,  CATE,  and 
CENPAC  Block  Scores 


suggested  that  students  were  only  slightly  more  comfortable  operating 
either  of  the  simulators  (3.6)  than  operating  the  actual  equipment 
(3*4).  Students  trained  on  the  2-D  simulator  estimated  that  more  of 
their  time  was  spent  on  troubleshooting  exercises  (3.0)  than  did 
students  in  either  the  3-D  or  AET  training  groups  (2.7  and  2.6  respec¬ 
tively).  While  all  students  believed  strongly  that  more 
troubleshooting  experience  was  necessary,  this  belief  was  somewhat 
more  evident  for  3-D  trained  students  (4.7)  than  for  students  taught 
with  the  2-D  simulator  (4.4)  or  AET  (4.1).  Students  trained  on  the  3- 
D  simulator  also  felt  less  prepared  for  their  field  assignments  in 
terms  of  troubleshooting  experience  (2.1)  than  did  students  in  the  2-D 
or  AET  training  modes  (2.4,  2.5).  Overall,  students  were  very 
supportive  of  their  instructors,  regardless  of  the  equipment  used  for 
training  (4.2).  A  complete  summary  of  student  interview  results  is 
provided  in  Section  A-5  of  Appendix  A  as  a  function  of  the  primary 
equipment  used  for  6883  training. 


Instructor  Interviews 


The  instructors  were  asked  to  -se  a  5-point  scale  to  rate 
simulator  training  in  general  on  several  factors  (e.g.,  Is  it  a  good 
idea?;  Can  it  provide  AET  equivalent  training?;  Is  it  more  reliable 
than  AET  training?)  based  on  their  experiences  with  the  two  simulators 
used  in  this  study.  Only  one  statement  received  a  below  average  (2.4) 
rating:  that  simulators  could  replace  actual  equipment  in  training. 
This  finding  was  not  surprising  since  previous  studies  have  shown 
instructors  strongly  believe  in  the  necessity  of  AETs  for  training 
(e.g.,  Fink  &  Shriver,  1978).  Statements  which  drew  strong  support 
from  the  instructors  included  (a)  simulation  can  provide  adequate 
training  at  a  cost  savings  (4.0),  (b)  simulators  can  provide  more 
variety  of  training  than  can  AETs  (4.2),  and  (c)  simulators  are  more 
reliable  than  actual  equipment  (4.6). 

When  asked  to  rate  13  elements  which  are  considered  important 
in  simulator  design,  six  were  given  especially  high  marks  by  the 
instructors.  These  were  (a)  software  should  be  suitable  to  the  course 
objectives  (4.7),  (b)  instructors  should  have  input  or  control  over 
software  development  (4.8),  (c)  the  Air  Force  should  have  control  over 
the  design  of  the  simulator  (4.4),  (d)  the  equipment  must  have  opera¬ 
tional  reliability  (4.9),  (e)  the  equipment  must  be  easy  to  maintain 
(4.6),  and  (f)  the  software  must  include  a  broad  variety  of  lessons 
(4.1).  The  least  important  element  of  the  13  was  the  inclusion  of  a 
student  performance  monitoring  capability  (3.1).  It  should  be  noted 
that  the  AET  does  not  have  such  a  capability,  and,  although  the  two 
simulators  are  able  to  track  student  performance,  this  capability  was 
used  sparingly  during  the  evaluation  period  by  instructors. 
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Instructors  were  also  asked  to  rate  a  number  of  statements 
that  might  apply  to  both  the  2-D  and  3-D  simulators.  A  comparison  of 
mean  rating  results  as  a  function  of  equipment  showed  that  instructors 
agreed  (a)  the  2-D  hardware  was  slightly  more  simplistic  (2.9)  than 
was  the  3-D  hardware  (2.0),  (b)  the  3-D  simulator  was  more  likely  to 
be  conceived  of  as  similar  to  the  AET  by  students  (3*1)  than  was  the 
2-D  simulator  (2.1),  and  (c)  when  compared  to  the  AET,  the  3-D  was  a 
somewhat  better  trainer  (2.8)  than  was  the  2-D  (2.1). 

The  impact  of  the  introduction  of  simulators  on  the  training 
regime  was  also  investigated  by  asking  the  instructors  to  what  extent 
their  experiences  with  the  simulators  affected  their  approach  to 
teaching.  Of  the  nine  respondents,  four  reported  they  had  identified 
weaknesses  in  the  training  materials  and  had  requested  changes  be  made 
in  these  materials,  and  four  reported  placing  more  emphasis  on  certain 
portions  of  the  course  material.  Complete  results  from  the  instructor 
interview  are  provided  in  Section  A-6  of  Appendix  A. 

The  instructor  survey  was  also  designed  to  determine  preference 
regarding  future  use  of  the  three  trainers  in  the  classroom.  Due  to  a 
poorly  constructed  survey  item,  however,  it  was  determined  that  the 
responses  concerning  future  use  were  confounded  by  prior  experience 
with  the  various  trainers.  Thus,  the  average  response  score  for  this 
item  was  not  included  in  the  analysis.  Since  this  issue  was  considered 
to  be  of  significant  importance  to  the  evaluation,  a  separate  interview 
was  conducted  with  the  same  instructors  to  determine  trainer  preferences. 
Instructor  preference  was  overwhelmingly  in  favor  of  the  AET.  Of  eight 
respondents,  six  ranked  trie  AET  as  their  first  choice,  with  the  remain¬ 
ing  two  responses  being  split  between  the  simulators.  The  2-D  simulator 
was  ranked  as  the  last  choice  (i.e.,  third)  by  five  respondents, 
followed  by  the  3-D  simulator  by  three  respondents,  and  the  AET  was 
ranked  as  the  last  preference  by  only  one  respondent.  Among  the  reasons 
given  for  these  choices  of  trainers,  the  one-to-one  relationship 
between  the  AET  and  the  expected  field  experience  was  most  often  cited, 
despite  recognition  of  the  poor  record  of  reliability  of  the  AET.  The 
choice  of  the  3-D  over  the  2-D  simulator  also  reflects  an  instructor 
preference  for  high  physical  fidelity,  which  they  perceive  to  reside 
with  the  3-D  simulator  rather  than  with  the  2-D  simulator. 


Assessment  of  Field  Performance 


To  determine  whether  technicians  trained  on  simulators  and  on 
AETs  were  equally  capable  of  operating  F— 1 1 1  automatic  test  stations 
in  the  field,  subjective  ratings  of  each  technician's  performance  were 
collected  from  the  technicians  themselves  and  from  their  supervisors. 
However,  because  of  the  12  week  time  lag  between  the  completion  of 
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6883  training  and  the  administration  of  field  follow  up  questionnaires, 
not  all  students  who  had  participated  in  the  performance  evaluation 
phase  were  available  for  follow  up  prior  to  the  finalization  of  this 
report.  The  abilities  of  those  students  assigned  to  6883  and  the 
highly  similar  6873  equipment  were  of  particular  interest,  although 
actual  assignments  to  these  stations  were  quite  low.  Only  10  of  the 
98  technicians  who  responded  to  the  field  survey  were  assigned  to  6883 
or  6873  test  stations.  For  this  reason,  it  was  not  possible  to 
reliably  assess  the  impressions  of  these  few  technicians  as  a  function 
of  each  of  the  three  6883  training  modes  they  might  have  encountered. 
The  results  discussed  in  the  following  sections,  therefore,  are  based 
on  the  entire  sample  of  98  technicians  and  their  supervisors  regardless 
of  equipment  assignments. 


Technician  Follow  Up  Survey 

Mean  ratings  for  selected  items  from  the  technician  survey  are 
presented  in  parentheses  throughout  this  discussion.  All  ratings  are 
based  on  a  5-point  scale  where  5  indicates  a  highly  positive  response. 

A  complete  summary  of  these  data  is  provided  in  Section  A-7  of  Appendix 
A.  It  should  be  noted  at  the  outset  that  the  respondents  generally 
believed  that  6883  training  had  been  relatively  unimportant  for  their 
current  field  assignments  (2.3);  this  finding  reflects  the  previous 
comments  regarding  equipment  assignments  and  supports  the  earlier 
conclusion  that  significant  training  effects  would  not  be  anticipated 
based  on  the  brief  6883  training  manipulation. 

The  results  showed  that  all  technicians  were  reasonably  com¬ 
fortable  operating  actual  field  equipment  and  that  this  was  perhaps 
more  true  for  3-D  trained  personnel  (4.1)  than  those  trained  with  2-D 
(3-8)  or  AET  (3.8)  equipment.  When  asked  about  their  ability  to  cable 
LRUs  and  adapters  to  the  test  stations,  the  AET  and  3-D  trained  tech¬ 
nicians  felt  slightly  more  confident  (3.1,  2.9)  than  did  those  who 
used  the  2-D  simulator  (2.6).  Technicians  also  believed  that  they 
were  somewhat  better  trained  in  the  use  of  common  test  equipment  if 
they  had  AET  training  (4.0)  than  if  they  had  simulator-based  training 
(3.7  for  3-D  and  3-6  for  2-D). 

With  respect  to  how  well  technicians  felt  they  had  been  pre¬ 
pared  for  field  work  in  terms  of  troubleshooting  skills,  the  responses 
were  quite  consistent  with  the  earlier  student  interview  results. 
Overall,  the  technicians  rated  ATC  troubleshooting  training  as  ade¬ 
quate;  less  so  for  3-D  training  (2.4)  than  for  other  training  modes 
(2.7). 


Those  technicians  who  had  been  trained  on  one  of  the  simulators 
were  asked  four  questions  pertaining  to  their  experience  with  simulated 
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training.  Technicians  trained  on  the  2-D  reported  receiving  more 
enjoyment  from  their  experience  than  did  3-D  trained  technicians  (3.5 
vs  3.2);  however,  3-D  and  2-D  trained  technicians  were  equally  likely 
to  rate  the  simulator  with  which  they  had  had  experience  as  an  adequate 
simulator  of  actual  test  station  operation  conditions  (3.3  for  3-D  and 
3.2  for  2-D).  Neither  group  was  supportive  of  the  idea  that  simulators 
were  better  training  instruments  than  were  AETs  (2.2  for  3-D  and  2.M 
for  2-D)  or  that  simulators  should  be  used  more  frequently  during  ATC 
training  (2.8  for  3-D  and  3.0  for  2-D). 


Supervisor  Survey 

The  results  of  the  supervisor  survey  revealed  no  substantial 
performance  differences  among  technicians  who  had  received  various 
modes  of  6883  training.  In  general,  the  ratings  were  high,  indicating 
that  supervisory  personnel  are  satisfied  with  the  abilities  of  elec¬ 
tronic  maintenance  technicians  currently  being  trained  by  ATC.  Section 
A-8  of  Appendix  A  summarizes  the  mean  ratings  given  for  selected  survey 
questions. 


Contextual  Analysis  of  Training  Environments 


Analyses  of  several  contextual  elements  were  included  in  this 
evaluation  effort.  These  elements  were  grouped  into  three  general 
areas:  system  capabilities,  system  support  requirements  and  operating 
characteristics,  and  equipment  reliability. 


System  Capabilities 

Each  of  the  three  systems  evaluated  (the  normal  operational 
configurations  of  the  AET,  the  3-D  simulator  and  the  2-D  simulator 
with  part-task  trainers)  has  different  capabilities  defining  operation 
and  training  utility  (Table  7).  Some  of  the  comparisons  of  particular 
interest  were  (a)  the  ability  to  insert  equipment  malfunctions,  (b) 
the  capability  for  providing  troubleshooting  experience,  (c)  the 
amount  of  student  contact  time  with  the  trainer,  (d)  safety,  and  (e) 
ease  of  maintenance  and  modification.  In  three  of  these  categories, 
the  2-D  system  was  rated  most  favorably;  in  two  of  these  areas  the  3-D 
and  2-D  were  rated  equally  capable.  The  AET  was  rated  low  in  several 
capabilities  areas  and  was  considered  relatively  difficult  to  maintain 
and  modify. 

The  capabilities  identified  and  compared  in  Table  7  do  not,  of 
course,  establish  which  system  was  most  effective  for  training 
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Training  Device 


Factor 

AET 

3-D  Simulator 

2-D  Simulator* 

Capabilities 

Malfunction  Insertion 

Very  Limited 

Unlimited+ 

Unlimited+ 

Troubleshooting 

Very  Limited** 

Limited++ 

Unlimited* 

Part-Task  Training 

Low 

Very  Low 

High 

Self-Paced  Learning 

Low 

High 

High 

Student  Contact 

Low 

Low 

High 

Student  Performance 
Monitoring 

No 

Yes 

Yes 

Support  Requirements  and  Operating  Characteristics 

Mobility 

Low 

Very  Low 

High 

Operating  Noise  Level 

High 

High 

Low 

Safety 

Low 

High 

High 

Power  Requirements 

1 15v,  60  Hz 

115v,  400  Hz 

120  VAC 

120  VAC 

Operational  Responsiveness 
to  Keyboard  Input 

Fast 

Slow 

Very  Slow 

Physical  Representativeness 
to  AET 

— 

High 

Moderate 

Operational  Reprer  entative- 
ness  to  AET 

— 

Moderate 

Moderate 

Ease  of  Maintenance 

Poor 

Fair 

Good 

Spare  Parts  Requirement 

Very  High 

Moderate 

Low 

System  Modifications 

Difficult 

Easy 

Easy 

System  Support  Equipment 

Extensive 

Minor 

Minor 

•includes  Part-Task  Trainers  +Restricted  only  by  lessonware 

••Restricted  to  unscheduled  development  limitations 

malfunctions  ++Nonexistence  of  patch  panel 
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purposes.  Performance  analyses  and  user  acceptance,  previously 
presented,  and  cost  elements  discussed  in  the  next  section  were  the 
key  elements  in  system  comparability. 


System  Requirements  and  Characteristics 

The  three  6883  training  devices  differed  considerably  with 
respect  to  support  requirements  and  operating  characteristics.  Table 
7  provides  a  summary  comparison  of  the  three  training  devices  in 
relation  to  these  factors.  Some  of  the  more  prominent  environmental 
differences  are  discussed  in  the  following  sections. 

Housing.  To  meet  the  combined  theoretical  and  practical  objec¬ 
tives  of  training,  all  6883  training  equipment  was  originally  housed 
in  rooms  that  included  tables  and  chairs  as  well  as  the  training  device. 
During  the  first  half  of  the  project  period,  the  three  units  (AET,  3- 
D,  and  2-D)  were  housed  in  similarly  configured  rooms  with  each  unit 
in  its  own  room.  The  AET  room  included  a  large  air  conditioning  unit 
(floor-mounted)  whereas  the  two  simulator  rooms  had  window-mounted  air 
conditioning  units.  Subsequently,  all  three  units  were  moved  to  a  new 
building  with  central  air  conditioning.  The  AET  and  3-D  simulator 
were  housed  in  the  same  room  in  this  new  facility.  The  room  was  quite 
large,  approximately  40'  by  25',  with  a  linoleum  floor  and  no  chalk¬ 
boards.  The  2-D  simulator  room  was  smaller  (30*  by  18')  in  size  but 
included  features  such  as  a  carpeted  floor,  a  chalkboard,  and  a  more 
permanent  arrangement  of  tables  and  chairs  for  instruction. 

In  reality,  the  specific  housing  requirements  for  the  three 
trainers  are  minimal.  All  trainers  require  air  conditioning  to  main¬ 
tain  optimum  operating  temperatures.  Space  requirements  for  the 
trainers  can  be  met  by  standard  classroom-size  areas,  assuming  the 
CENPAC  computers  continue  to  be  housed  in  quarters  separate  from  the 
AET,  as  is  currently  the  case.  Additional  space,  furnishings,  or 
facilities  serve  primarily  to  enhance  the  human  and  instructional 
environment  (e.g. ,  carpeting,  chalkboard). 

Support.  To  operate  each  of  the  three  devices,  certain  support 
requirements  must  be  met.  For  the  AET,  these  include  high  electrical 
power  inputs,  air  conditioning,  adjunct  CENPAC  computers  (which  serve 
all  operating  test  stations),  and  a  substantial  number  of  spare  elec¬ 
tronic  parts.  The  3-D  and  2-D  simulators  require  only  standard  120 
volts  AC  wall  outlet  power  for  operation.  The  computers  which  operate 
these  simulators  are  exclusive  to  the  units  and  the  simulator  systems 
have  need  for  few  replacement  parts.  Air  conditioning  is  provided  to 
each  classroom  via  a  central  system. 
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Safety.  Because  of  the  high  power  requirements  of  the  AET, 
safety  was  an  important  consideration  in  the  training  environment. 
Potential  safety  problems  are  aggravated  by  the  existence  of  several 
TRU  and  LRU  connectors,  jacks,  and  other  points  of  possible  electrical 
power  output  from  the  AET.  Both  simulator  systems,  with  their  lower 
power  requirements  and  fewer  potential  contact  points,  reduce  the 
safety  problems  considerably. 

Noise  Levels.  The  noise  level  of  an  operating  test  station  is 
a  significant  environmental  factor  for  two  primary  reasons:  the  volume 
and  frequency  of  the  noise  level  is  often  physically  irritating,  and 
the  noise  level  interferes  with  instructor-student  communication. 
Special  efforts  were  made,  therefore,  to  record  actual  operational 
noise  levels  for  each  device  in  order  to  substantiate  these  observa¬ 
tions.  The  results  are  provided  in  Table  8.  The  high  level  of  noise 
attributed  to  the  operating  AET  is  due  to  the  fact  that,  in  its  normal 
operation,  the  6883  AET  requires  the  operation  of  a  blower  unit  in  ad¬ 
dition  to  air  conditioning  required  to  dissipate  heat  from  the  TRUs 
and  LRUs.  The  3-D  system  is  also,  though  to  a  lesser  degree  than  the 
AET,  a  noisy  system  in  operation.  This  is  due  to  the  existence  of  two 
computers,  blower  units  in  the  computer  racks,  and  required  air 
conditioning.  The  2-D  simulator  itself  generates  virtually  no  noise; 
however,  the  2-D  system  also  requires  an  air  conditioning  system  to 
maintain  operational  temperatures.  Nonetheless,  the  2-D  operational 
environment  is  measurably  quieter  than  the  operational  environments  of 
the  AET  or  the  3-D  system. 

Table  8  shows  (a)  the  values  recorded  with  equipment  and  back¬ 
ground  noise  together  and  (b)  the  adjusted  values  which  reflect 
equipment-only  noise.  This  adjustment,  using  a  standard  logarithmic 
formula,  was  necessitated  by  the  fact  that  background  noise  could  not 
be  eliminated  from  the  environment.  For  the  specific  formula  used  to 
determine  the  equipment-only  values,  see  Appendix  C.  In  normal  class¬ 
room  usage,  air  conditioning  (the  primary  source  of  background  noise) 
for  each  training  device  is  required  to  maintain  operable  temperature 
levels.  Thus,  the  values  in  the  upper  portion  of  the  table  reflect 
normal  classroom  noise  levels. 

Since  the  2-D  simulator  is  operated  in  a  classroom  environment 
that  includes  carpeting,  tables,  chairs,  and  other  sound-absorbing 
elements,  additional  readings  were  taken  in  this  classroom  environment. 
As  shown  in  Table  8,  the  overall  unadjusted  noise  level  was  66  dBa  at 
5  feet,  with  a  background  noise  level  of  only  62.5  dBa.  The  adjusted 
noise  level  in  this  environment  was  63. ^  dBa. 

The  accuracy  of  the  readings,  within  +1  dBa,  is  sufficient  to 
ascertain  that  all  are  within  Occupational  Safety  and  Health  Agency 
guidelines  for  nondamaging  noise  levels  for  an  eight  hour  period. 
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Table  8 


Noise  Level  Readings  for  the  Three 
6883  Training  Devices  (in  dBa)1 


Equipment 

Distance 
from  Equipment 

AET  3-D 

2-D 

2-D2 

Equipment  and  Background  Noise3 

Near  (1  ft.) 

77.50  73.90 

67.00 

- 

Intermediate  (5 

ft.) 

77.83  73.00 

66.00 

66.00 

Distant  (10  ft. 

) 

75.83  71.50 

65.88 

- 

Equipment  Noise  Only 

Near  (1  ft.) 

77.2  73.2 

62.7 

- 

Intermediate  (5 

ft.) 

77.6  72.3 

59.1 

63.4 

Distant  (10  ft. 

) 

75.4  70.4 

58.6 

- 

1  All  readings,  unless  otherwise  stated,  were  taken  in  the 
laboratory  room  which  normally  houses  the  AET  and  3-D  trainers. 

^Readings  in  the  classroom  housing  the  2-D  where  a  background 
noise  level  of  62.5  dBa  was  measured.  Readings  were  taken  at  one 
distance  since  movement  toward  and  away  from  the  unit  produced  no 
significant  fluctuations  in  the  meter  reading 

^Background  noise  was  measured  at  65  dBa 


Environmental  Protection  Agency  standards  suggest  that  to  conduct 
training  using  a  normal  voice  with  the  sound  levels  measured  in  the 
normal  classroom  environments  experienced  in  this  study,  the  instructor 
could  be  no  more  than  .5  foot  (for  the  higher  measurement  of  the  AET 
environment)  to  2  feet  (for  the  lower  measurement  of  the  2-D  environ¬ 
ment)  from  the  students.  Using  a  raised  voice  (or  a  "classroom"  voice), 
appropriate  distances  would  be  1  foot  (AET)  to  3  feet  (2-D).  Tables 
indicating  interference  levels  for  various  dBa  levels  and  distances 
are  included  in  Appendix  C. 


Equipment  Reliability 

A  fundamental  issue  involved  in  the  evaluation  of  simulated 
training  devices  is  training  equipment  reliability.  Equipment  relia¬ 
bility  has  potential  impacts  on  both  performance  and  cost  and,  in  large 
part,  defines  the  format  of  the  training  approach  used  with  actual 
test  station  equipment.  Unlike  the  approach  with  preprogrammed  mal¬ 
functions  on  the  simulators,  the  acquisition  of  practical  experience 
on  the  actual  equipment  is  dependent  on  pre-existing  or  unexpected 
equipment  failures.  This  is  particularly  true  of  the  troubleshooting 
aspects  of  training.  In  extreme  cases  (which  were  observed  during 
this  evaluation),  this  dependency  results  in  very  limited  training. 

For  example,  when  all  TRUs  and  LRUs  operate  without  malfunction,  it  is 
not  possible  to  demonstrate  troubleshooting  techniques.  At  other  times 
when  a  specific  TRU  failure  causes  the  test  station  to  be  inoperative, 
no  training  is  possible,  or  extensive  attention  to  one  portion  of  the 
test  station  results. 

The  observed  reliability  of  the  6683  test  station,  the  3-D 
simulator  and  the  2-D  simulator  during  the  data  collection  period  is 
graphically  depicted  in  Figure  11.  It  can  be  seen  that  the  AET  was 
inoperable  for  a  6  month  period.  The  primary  cause  of  the  downtime  on 
the  AET  was  the  relocation  of  F-111  Test  Stations  to  a  new  training 
facility.  Not  only  was  the  AET  unavailable  for  use  during  the  period 
of  the  move,  but  once  the  device  was  in  place,  it  became  apparent  that 
the  new  electrical  system  was  insufficient  for  proper  AET  operation. 
While  the  AET  had  been  experiencing  serious  malfunctions  prior  to  the 
move  and  it  was  possible  to  make  limited  repairs  without  station  power, 
maintenance  efforts  were  severely  restricted  until  the  building  power 
problems  were  solved.  Ill-timed  preventive  maintenance  also  caused 
some  difficulty  for  assessing  reliability  of  the  AET.  Specifically, 
an  LRU  cable  which  was  necessary  for  both  AET  training  and  the 
contractor’s  hands-on  testing  was  sent  off-base  for  refurbishment 
during  the  same  general  time.  This  action  further  contributed  to  the 
unavailability  of  the  equipment  for  training. 
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Evaluation  Project  Data  Collection  Period 
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Figure  11.  Equipment  reliability  during  the  data  collection  period 


The  3-D  simulator  experienced  three  periods  of  downtime 
primarily  due  to  interface  problems  between  the  two  computers  associ¬ 
ated  with  the  3-D  simulator  system.  One  of  these  periods  occurred 
directly  after,  and  most  likely  as  a  result  of,  the  relocation  of  the 
3-D  simulator  to  the  new  training  facility.  Other  periods  occurred 
before  the  equipment  move  and  near  the  end  of  the  evaluation  period. 

The  2-D  simulator  experienced  no  significant  downtime  and  was 
moved  to  the  new  facility  without  apparent  complications.  The  Patch 
Panel  Trainer  experienced  only  1  day  of  downtime.  Since  the  other 
part-task  trainers  were  either  inoperable  (i.e.,  Logic  Trainer)  or 
rarely  used  (i.e.,  Oscilloscope  Trainer),  their  reliability  remains  to 
be  demonstrated.  The  results  of  this  analysis  suggest  that  the  two 
simulators,  and  especially  the  2-D  simulator,  were  highly  reliable 
trainers  in  comparison  to  the  AET. 

The  reliability  of  the  trainers  influenced  the  evaluation 
effort  in  at  least  two  ways.  First,  the  frequency  and  duration  of 
individual  equipment  failures  shown  in  Figure  11  were  sufficient  to 
cause  disruption  of  project  training  assignments  and  data  collection. 
The  major  impact  of  malfunctions  on  the  evaluation  design  was  on  the 
assignment  of  trainees  to  experimental  groups.  Thus,  flexibility  in 
the  evaluation  design  was  essential  to  allow  for  unexpected  changes  in 
the  training  schedule  and  to  minimize  the  potential  loss  of  performance 
data.  Second,  only  minimal  training  time  was  lost  due  to  the  avail¬ 
ability  of  the  three  training  instruments.  For  example,  all  AETs 
associated  with  F-111  electronics  maintenance  training  were  unavailable 
for  training  or  testing  during  the  data  collection  period  associated 
with  the  power  source  problems.  Only  the  6883  block  of  instruction 
did  not  require  issuance  of  "training  deviations,"  since  at  least  one 
of  the  simulators  was  operational  throughout  that  period. 

The  existence  of  alternative  training  devices  can  significantly 
affect  the  assessment  of  equipment  reliability  in  the  training  environ¬ 
ment.  For  example,  in  the  present  study,  there  appeared  to  be  little 
motivation  to  effect  immediate  repairs  on  a  particular  device  since 
there  was  always  at  least  one  suitable  alternative  device  available 
for  training  purposes.  Thus,  even  when  parts  were  available  to  effect 
repairs  on  a  malfunctioning  trainer,  the  availability  of  an  alternative 
device  lessened  the  motivation  to  complete  those  repairs. 

Alternative  methods  are  available  for  determining  equipment 
reliability.  For  example,  downtime  can  be  defined  as  the  time  to  make 
repairs,  excluding  the  time  spent  waiting  on  parts.  This  approach 
eliminates  the  problems  of  specifying  all  factors  contributing  to  down¬ 
time,  but  does  not  accurately  reflect  the  duration  of  downtime  periods. 
A  better  measure  may  be  the  operating  time  between  repairs.  This 
measure  is  unrealistic,  however,  when  applied  to  ATC  since  its  low 
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priority  for  replacement  parts  can  result  in  periods  of  test  station 
inoperability  exceeding  1  year,  thus  creating  the  possibility  of 
recording  few  or  no  periods  of  equipment  operation  during  an  evaluation 
study.  A  measure  combining  time  to  repair  and  time  awaiting  parts  is 
more  realistic.  While  this  measure  can  also  be  artifically  inflated 
due  to  the  low  priority  on  parts,  it  was  used  in  this  study  since  these 
two  elements  best  typify  the  real  training  environment  and  serve  as 
indicators  of  system  maintainability  (i.e.,  frequency  and  duration  of 
malfunctions,  and  availability  of  spares). 


Cost  Analysis 


Table  9  shows  the  major  factors  of  the  life  cycle  cost  compari¬ 
son  among  training  devices.  Recurring  and  nonrecurring  costs  have 
been  combined  to  provide  profiles  of  costs  in  constant  1978  dollars. 

The  NPV  of  each  cost  stream  projected  for  15-year  life  cycles  and 
discounted  at  10  percent  is  also  shown,  along  with  the  estimated 
training  costs  per  student-hour  of  instruction. 

As  can  be  seen  from  Table  9,  the  differences  in  total  costs 
can  be  attributed  almost  entirely  to  differences  in  the  costs  of 
purchasing  and  sustaining  the  equipment.  Personnel  and  instructional 
materials  costs  are  quite  comparable  across  the  three  training  devices. 
Facilities  and  student  costs  are  identical  for  each  trainer  since  equal 
training  interventions  were  assumed.  Alternative  cost  scenarios  are 
clearly  possible,  however,  in  which  student  costs  would  be  expected  to 
vary  as  a  function  of  training  provided.  The  final  entries  of  Table  9 
show  the  average  cost  per  student  hour  of  instruction  based  on  the 
total  NPV  of  each  device.  Both  simulators  show  a  substantial  savings 
over  the  average  cost  associated  with  AET  training. 

Tables  10,  11,  and  12  show  individual  estimates  for  cost 
elements  that  comprise  each  category,  together  with  explanatory  infor¬ 
mation  for  the  AET,  the  3-D  simulator  and  2-D  simulator,  respectively. 
The  tables  are  structured  so  that  the  1978  Investment  Costs  can  be 
considered  as  sunk  costs;  from  this  perspective,  the  costs  in  constant 
dollars  of  operating  the  AET  and  the  simulators  for  15-year  life  cycles 
are  $3,366,150  (AET),  $1,588,020  (3-D)  and  $1,294,410  (2-D).  The  life 
cycle  cost  comparison,  which  clearly  favors  the  simulators,  and  the  2- 
D  simulator  in  particular,  establishes  that  simulation  is  an  important 
option  for  maintenance  training.  The  following  examination  of  some  of 
the  cost  assumptions  inherent  in  this  comparison  further  supports  this 
finding. 
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Life  Cycle  Costs  (Thousands  of  Dollars) 
or  AET  Simulator — Constant  1978  Dollars 


Total  cost  In  constant  dollars  =  $5,335,980 
Net  present  value  (1978)  =  $3,760,680 


Life  Cycle  Costs  (Thousands  of  Dollars) 
for  3-D  Simulator — Constant  1978  Dollars 
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Total  cost  in  constant  dollars  =  $2,183.00 
Net  present  value  (1978)  =  $1,501.09 


mulator — Constant  1978 


Miscellaneous  Not  estimated  No  supplies  regularly  used  in 

_ 6883  simulation  training _ 


Cost  Considerations 


A  major  issue  involved  in  planning  this  analysis  was  that  the 
acquistion  cost  of  each  simulator  system  was  estimated  by  project 
administrators  to  be  higher  than  is  represented  by  the  actual  contract 
cost.  Only  now  can  this  concern  be  put  into  proper  perspective,  and 
the  question  can  be  asked,  "What  is  the  impact  on  the  cost  effective¬ 
ness  ratio  if  the  actual  acquisition  cost  of  the  simulator  is  perhaps 
double  the  contract  price?"  In  an  NPV  calculation,  the  acquisition 
costs  are  not  discounted;  thus,  the  effect  of  doubling  the  contract 
price  is  not  obscured  by  the  discounting  process.  For  example,  if  the 
actual  cost  of  the  3-D  simulator  were  doubled,  increasing  the  NPV  of 
the  simulator  to  $2,049,090,  the  cost  per  student-hour  of  instruction 
would  increase  only  39  percent  ($32/hour),  just  over  one-half  of  the 
cost  found  for  the  AET.  Similarly,  if  the  actual  cost  of  the  2-D 
simulator  were  doubled,  the  NPV  would  be  $1,374,330.  The  cost  per 
student-hour  would  increase  only  26  percent  ($21 /hour),  still  only 
about  one-third  the  cost  found  for  the  AET. 

Further,  the  LSC  model  developed  by  the  Sim  SPO  provides 
additional  insight  into  the  relationship  between  system  cost  and  the 
Sustaining  Investment  cost  component,  which  involves  test  station 
spares,  spare  components,  and  inventory  management.  This  model,  used 
to  assess  costs  of  the  F— 1 6  simulator,  indicated  that  the  initial 
system  acquisition  and  support  costs  ($4,919,000  and  $4,965,000, 
respectively)  were  approximately  equal  to  the  15-year  life  cycle  costs 
projected  for  "Sustaining  Investment"  (Eggemeier,  1979,  Table  6). 

This  relationship  is  important  because  it  reflects  the  lower  cost  of 
spares  associated  with  simulators  based  on  actual  cost  experience  with 
one-of-a-kind  systems.  Table  11  presents  an  estimate  for  Sustaining 
Investment  over  the  life  cycle  of  the  6883  3-D  simulator.  The  $941,330 
projected  is  relatively  consistent  with  an  estimate  of  about  $1,100,000 
for  system  and  support  costs  of  the  simulator  (double  the  contract 
value).  Table  12  shows  that  the  projected  Sustaining  Investment  of 
$694,210  for  the  2-D  simulator  is  also  consistent  with  an  estimated 
$640,000  (double  the  contract  value)  for  system  and  support  costs  (see, 
however,  Note  Q,  Appendix  D). 

In  considering  the  policy  implications  of  a  2.5  to  1  ratio  in 
the  cost  effectiveness  of  the  3-D  simulator  and  a  3.6  to  1  ratio  for 
the  2-D  simulator  as  each  is  compared  to  the  AET  (rf.  Table  9),  it  is 
important  to  understand  that  the  estimate  for  the  AET  is  extremely 
conservative.  It  was  noted  earlier  that  the  costs  of  the  CENPAC 
computers  were  not  allocated  to  the  AET.  Examination  of  Table  10 
shows  that  no  cost  element  was  included  in  the  equipment  category  that 
reflects  the  cost  of  installation  and  start-up  of  the  AET.  In  contrast, 
installation  and  start-up  costs  for  the  simulators  were  fully  allocated 
as  shown  in  Tables  11  and  12.  Start-up  costs  for  the  3-D  simulator 
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were  unexpectedly  high  due  to  problems  encountered  in  bringing  the 
simulator  on-line. 

The  particular  value  of  the  cost  comparison  lies  in  the  notion 
that  with  cost  experience  data  derived  from  actual  trainer  alternatives, 
there  is  an  obvious  cost  of  ownership  preference  for  simulators.  Table 
9  also  shows  that  the  2-D  simulator  is  somewhat  more  cost  effective 
than  is  the  3-D  simulator.  Although  there  was  a  cost  savings  in  devel¬ 
oping  the  specifications  for  the  2-D  simulator  due  to  the  existence  of 
the  3-D  simulator,  the  major  difference  lies  in  Acquisition  Costs  and 
Sustaining  Investment  costs. 

It  is  important  to  make  comparisons  between  the  two  6883 
simulators  cautiously,  however.  First,  both  devices  are  prototypes 
and  each  was  designed  to  test  the  feasibility  of  a  different  config¬ 
uration.  The  cost  of  the  3-D  simulator  was  most  certainly  higher  due 
to  intent  to  drive  four  independent,  complete  student  stations. 
Elimination  of  this  device  specification  would  have  reduced  costs 
considerably  since  only  one  computer  system  would  have  been  needed. 
Similarly,  the  part-task  trainer  approach  employed  in  the  design  of 
the  2-D  simulator  provides  an  opportunity  to  train  four  students 
simultaneously  on  different  aspects  of  the  6883  system.  To  date, 
however,  two  of  the  task  trainers  have  not  been  consistently  employed 
by  instructors  in  the  classroom.  Even  though  student  performance,  as 
measured  by  the  tests  utilized  in  the  present  study,  is  equivalent  on 
all  three  training  devices,  it  is  not  clear  that  the  three  trainers 
provide  equivalent  training  opportunities.  It  must  be  noted  that  the 
simulators  were  incorporated  into  the  existing  curriculum  and  no 
strategies  were  employed  to  maximize  the  training  capabilities  of 
either  simulator  device.  Any  comparison  of  the  costs  between  sim¬ 
ulators  must  be  based  on  the  assumption  of  training  equivalency  to  the 
AET  and  untapped  training  device  capabilities.  Indeed,  a  study  of  the 
unique  capabilities  of  each  device  is  recommended. 


Alternative  Simulator  Applications 

A  cost  effectiveness  advantage  has  been  determined  for  the 
6883  simulators  under  the  assumption  of  equal  training  effectiveness 
and  identical  patterns  of  trainer  use.  This  determination  has  partic¬ 
ular  significance  if,  in  fact,  the  simulator  does  provide  student 
training  to  the  same  level  as  achieved  on  the  AET.  As  already  dis¬ 
cussed,  while  no  major  performance  differences  were  observed,  it  is 
not  true  that  the  content  of  training  and  use  patterns  were  identical 
for  all  groups  studied. 

Figure  12  illustrates  the  relative  training  capabilities  of 
alternative  training  devices.  The  diagram  indicates  that  the  6883 
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Figure  12.  Training  capability  of  equipment  in  relation  to  the  implicit  hierarchy  of  training 
objectives. 


simulators  effectively  train  procedures  and  troubleshooting  through 
the  automatic  testing  features  of  the  device  to  a  level  consistent 
with  the  STS  for  3-level  personnel.  The  broken  line  associated  with 
the  AET  indicates  that  the  AET  trains  to  the  3-level  less  well  because 
of  the  unpredictable  nature  of  equipment  performance  during  each 
training  session  (the  uniformity  of  training  issue).  Although  the 
simulators  can  train  consistently  to  the  3-level,  the  AET  can  do  some¬ 
thing  that  cannot  be  done  on  the  simulators;  provide  hands-on  trouble¬ 
shooting  experience  when  the  fault  cannot  be  corrected  through 
automatic  testing  procedures.  The  diagram  shows  that  if  the  technical 
school  is  to  provide  experience  in  the  manual  troubleshooting  domain 
(regardless  of  how  well  such  requirements  are  reflected  in  the  STS), 
then  AETs  are  essential  to  the  training.  The  two  6883  simulators  have 
not  filled  this  gap  in  training  requirements,  although  another  type  of 
simulator  that  provides  manual  troubleshooting  training  might  be 
developed.  With  this  hierarchy  of  training  objectives  in  mind,  the 
cost  model  can  be  used  to  examine  alternative  simulation  interventions 
in  maintenance  training. 

A  generic  test  station  simulator.  A  review  of  the  operating 
status  of  the  10  test  stations  used  in  F-111  maintenance  training  sug¬ 
gested  (a)  that  manual  troubleshooting  experience  cannot  always  be  in¬ 
cluded  in  practical  exercise  sessions  due  to  variations  in  test  station 
availability  and  (b)  that  simulators  patterned  after  the  6883  could  be 
useful  to  deliver  training  related  to  procedure  and  automatic  fault 
isolation.  From  this  assessment,  it  could  be  argued  that  test  stations 
with  the  best  records  of  operational  status  are  in  a  real  sense  now 
being  used  as  generic  test  stations — to  the  extent  that  experience 
with  procedures  and  fault  isolation  must  be  transferable  to  other 
(nonoperating)  test  stations. 

Consider  the  possibility  of  using  a  generic  test  station 
simulator  as  a  means  of  providing  training  in  troubleshooting  proce¬ 
dures,  assuming  that  these  procedures  employ  only  automatic  testing 
for  fault  isolation.  The  pattern  of  use  suggested  for  this  generic 
simulator  is  one  in  which  lessons  are  tailored  to  meet  the  full  range 
of  F-111  maintenance  training  needs.  That  is,  the  available  set  of 
lessons  can  be  used  to  replace  training,  in  a  generic  sense,  that  would 
have  been  provided  on  actual  equipment,  if  it  was  operative.  Used  in 
this  way  the  generic  simulator  would  eliminate  the  need  for  AET 
training  on  some  of  the  10  trainers  now  used  in  the  course. 

The  cost  effectiveness  of  such  a  generic  simulator  can  be  ex¬ 
amined  in  a  straightforward  way.  A  cost  savings  will  not  necessarily 
be  realized  because  the  simulator  represents  an  additional  equipment 
cost.  In  fact,  increased  training  costs  will  be  incurred  because  the 
pattern  of  simulator  use  suggested  does  not  disturb  the  life  cycle 
cost  estimate  for  the  AETs.  In  a  life  cycle  cost  analysis  of  the  AET 
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training  system,  however,  if  the  costs  associated  with  student  and 
instructor  time  are  aggregated  for  all  procedural  and  fault  isolation 
training  time  lost  due  to  AET  status,  then  it  is  conceivable  that  the 
benefits  of  the  generic  simulator  will  exceed  costs.  If,  for  example, 
as  few  as  120  hours  per  student  are  lost  out  of  approximately  930  hours 
required  for  the  course  ( 1 3% ) *  then  a  simulator  would  be  justified  on 
the  basis  of  its  cost/benefit  ratio  over  the  15-year  life  cycle  since 
increased  training  effectiveness  would  offset  the  higher  training 
costs. 


Combined  test  station/simulator  complements.  A  second  approach 
to  using  simulators  in  F-111  maintenance  training  might  be  to  develop 
a  full  complement  of  simulators,  one  for  each  of  the  AETs.  The  sim¬ 
ulators  could  be  used  to  train  procedures  and  automatic  fault  isolation 
while  the  AETs  would  be  used  to  provide  equipment  familiarity  and 
"manual"  troubleshooting  experience.  The  cost  effectiveness  of  this 
approach  cannot  be  adequately  evaluated  without  more  information  on 
the  life  cycle  cost  of  the  current  training  system  and  some  estimate 
of  how  the  6883  AET  actually  compares  with  other  test  stations.  Never¬ 
theless,  it  is  unlikely  that  the  combination  of  reduced  training  time 
and  AET  maintenance  costs,  together  with  increased  manual  trouble¬ 
shooting  skills,  would  ever  result  in  a  cost/benefit  ratio  less  than 
one.  The  only  major  source  of  cost  saving  that  has  the  potential  of 
offsetting  the  simulator  investment  is  to  return  some  of  the  AETs  (and 
LRUs)  to  inventory  to  serve  as  spares  for  the  flight  line.  Based  on 
the  cost  effectiveness  ratio  found  for  the  6883  simulators,  it  would 
be  necessary  to  return  perhaps  four  or  five  AETs  to  F-111  weapon  system 
inventory;  training  for  those  blocks  of  instruction  affected  would 
then  be  accomplished  on  simulators.  Without  attempting  to  genet alize 
further,  this  option  may  be  worthy  of  consideration  by  ATC  given  the 
existing  AET  operational  status. 

Replace  AETs  with  simulators.  This  final  approach  to  employing 
simulators  in  maintenance  training  is  potentially  the  most  revolutionary 
one  because  it  suggests  new  patterns  in  simulator  use  and  simulator 
development.  It  proposes  that  all  of  the  AETs  are  returned  to  inventory 
and  that  no  attempt  be  made  to  train  manual  troubleshooting  skills  on 
actual  equipment  within  the  residence  phase  of  F-1 1 1  maintenance  train¬ 
ing.  Manual  troubleshooting  principles  could,  of  course,  be  taught 
using  advanced  simulators.  There  are  many  difficulties  inherent  in 
this  approach,  principally  because  implementation  would  require  changes 
both  in  the  residential  and  OJT  environments,  as  well  as  changes  in 
training  policy.  For  example,  the  residence  training  environment  would 
need  to  be  modified  to  optimize  the  effectiveness  of  simulators  in 
teaching  procedures  and  troubleshooting,  and  the  OJT  environment  would 
need  to  be  reorganized  to  insure  that  transfer  of  training  to  the 
actual  equipment  occurs. 
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One  outcome  of  using  simulators  in  this  way  might  be  a  reduc¬ 
tion  in  course  duration — possibly  to  16  weeks  from  23  weeks — in  view 
of  the  increased  training  potential  associated  with  the  simulators. 

It  must  be  recognized  that  students'  manual  troubleshooting  skills  are 
not  well  developed  in  the  training  environment  and  that  simulation 
technology  could  lead  to  a  redefinition  of  the  training  process. 

Evaluation  of  this  approach  to  simulator  training  is  not 
possible  at  this  time  because  of  the  broad  implications  for  OJT,  Air 
Force  Logistics  Command,  and  ATC.  It  has  been  presented  here  only  to 
show  the  range  of  options  that  might  be  considered  in  an  attempt  to 
establish  cost  effective  training  programs  that  maximize  the  potential 
of  simulated  equipment. 
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VI.  GENERAL  DISCUSSION 


The  results  of  this  study  indicate  that  (a)  both  the  3-D  and 
2-D  simulators  can  provide  training  at  least  equivalent  to  that 
currently  provided  by  the  AET,  (b)  these  simulators  can  be  expected  to 
yield  a  substantial  cost  savings  when  compared  to  the  cost  of  actual 
equipment  over  a  15-year  life  cycle,  (c)  the  simulators  are  more 
reliable  and  safer  training  devices  than  are  AETs,  and  (d)  the  sim¬ 
ulators  can  provide  more  consistent  and  controlled  training  across 
classes  than  can  AETs,  particularly  with  regard  to  troubleshooting 
training. 

This  study  reflects  the  use  of  the  simulators  in  an 
operational  avionics  maintenance  course  with  all  of  its  constraints. 
This  section  presents  a  discussion  of  the  potential  of  simulators  when 
employed  to  maximize  several  key  elements  of  instruction  in 
electronics  maintenance  training.  Given  the  findings  of  this  study, 
the  question  of  future  design  and  use  of  simulated  devices  in  the 
training  environment  remains.  Clearly,  the  decision  to  employ 
simulators  is  dependent  on  the  training  needs  identified. 


Need  for  More  Reliable  Trainers 


If  a  primary  concern  of  the  Air  Force  is  the  reliability  and 
safety  of  training  equipment,  then  this  study  suggests  that  simulators 
are  the  clear  choice.  Both  simulators  were  more  consistently 
available  for  training  and  necessitated  fewer  training  deviations  than 
did  AET  training.  Further,  the  reliability  of  the  2-D  simulator  was 
far  superior  to  that  of  the  3-D  simulator.  Assuming  that  experience 
on  maintenance  test  stations  is  an  essential  component  of  training, 
these  high  fidelity  simulators  serve  as  an  excellent  alternative  to 
actual  equipment.  It  is  important  to  note,  however,  that  the  high 
level  of  physical  and  psychological  fidelity  incorporated  into  the  3-D 
and  2-D®  simulators  in  this  study  makes  them  largely  unsuitable  for 
training  in  other  instructional  blocks.  Thus,  to  realize  the 
objectives  of  increased  reliability  and  safety  throughout  the  training 
course  it  would  be  necessary  to  develop  a  simulated  version  of  each 
test  station  used  for  training.  It  is  certainly  questionable  if  such 
an  approach  could  ultimately  be  considered  cost  effective,  especially 
if  the  simulators  are  planned  to  supplement  rather  than  replace  AETs. 


®It  is  recognized  that  some  components  of  the  2-D  simulator 
were  designed  to  be  used  as  generic  trainers.  However,  even  these 
components  of  the  simulator  had  considerable  physical  and 
psychological  fidelity  with  those  components  of  actual  test  stations. 
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Need  for  Less  Expensive  Training  Devices 


Clearly,  the  simulators  were  less  expensive  to  purchase  than 
actual  equipment,  and  the  increased  reliability  made  them  less  expen¬ 
sive  to  maintain.  Based  on  the  cost  data  alone,  it  seems  that  sim¬ 
ulators  are  the  preferred  training  devices.  Unfortunately ,  the  inter¬ 
pretation  of  this  cost  information  is  meaningless  without  considering 
the  overall  context  in  which  simulators  will  be  used.  If  no  actual 
equipment  is  available  for  training,  then  simulators  would  be  the  least 
expensive  training  devices  to  acquire.  However,  if  actual  equipment  is 
considered  essential  to  adequate  training,  and  the  same  number  of 
actual  units  would  be  purchased  for  this  purpose  regardless  of  a 
decision  to  employ  simulators,  then  the  cost  of  the  simulators  must  be 
considered  an  added  cost  of  training,  not  a  replacement  cost. 

There  is  some  evidence  from  this  study  that  high  fidelity  simu¬ 
lators  may  be  considered  acceptable  replacements  for  actual  equipment 
by  instructors.  These  devices  are  clearly  among  the  more  expensive 
simulated  trainer  options  available.  However,  the  potential  for 
replacing  actual  equipment  may  easily  offset  their  higher  acquisition 
and  maintenance  costs,  making  high  fidelity  simulators  the  most  cost 
effective  alternative  presently  available.  This  is  not  to  say  that 
low  fidelity  simulators  are  not  viable  training  devices.  Rather, 
there  is  a  greater  need  to  demonstrate  their  training  effectiveness  in 
order  to  secure  user  acceptance. 

In  the  event  that  the  use  of  simulators  would  reduce  the  number 
of  AETs  required,  the  cost  savings  of  simulators  must  be  calculated  by 
comparing  additional  copies  of  actual  equipment  with  the  cost  of  an 
initial  simulator.  Beyond  the  first  simulator,  cost  savings  could  be 
determined  by  comparing  the  cost  of  additional  copies  of  each  device. 

To  use  cost  appropriately  as  a  primary  factor  in  decisions 
concerning  future  acquisitions,  the  value  of  the  increased  reliability 
and  improved  maintainability  of  the  simulators  should  also  be  calcu¬ 
lated. 


Need  for  More  Consistent  Training 


Closely  related  to  the  fact  that  the  sir  lators  are  more 
reliable  than  actual  equipment  is  the  advantage  of  providing  consistent 
training.  Actual  equipment,  employed  as  trainers,  often  introduce 
digressions  into  the  training  regime  due  to  malfunctions.  When  this 
equipment  becomes  inoperative,  the  instructor  must  either  troubleshoot 
the  test  station  as  a  class  demonstration  or  go  on  to  teach  other 


aspects  of  the  course  which  do  not  require  operational  equipment. 

Should  a  decision  be  made  to  troubleshoot  the  equipment  "on  stage,"  a 
number  of  risks  are  involved.  First,  the  malfunction  may  be  unrelated 
to  the  material  being  taught.  Second,  the  problem  may  be  quite  complex 
and  thus  require  that  all  the  time  allocated  for  student  "hands-on" 
experience  be  spent  observing  the  instructor.  While  this  experience 
can  be  instructional,  it  often  is  not,  since  the  instructor's  objective 
becomes  repairing  the  test  station  rather  than  teaching  troubleshooting 
skills  to  the  class.  Third,  using  naturally  occurring  malfunctions  as 
instructional  experiences  can  be  both  frustrating  and  embarrassing  for 
an  instructor,  especially  when  the  problem  cannot  be  resolved  in  the 
available  class  time.  Finally,  when  repairing  the  test  station 
requires  ordering  parts,  the  equipment  often  remains  unavailable  for 
further  training  for  long  periods  of  time. 

The  simulators,  on  the  other  hand,  provide  a  pool  of  known 
malfunction  problems  which  can  be  used  to  teach  various  aspects  of  the 
equipment  in  a  planned  and  structured  manner.  Lesson  difficulty  can 
be  controlled  and  the  instructor  can  more  easily  focus  on  assessing 
student  troubleshooting  skills.  In  short,  the  simulators  offer  an 
opportunity  to  integrate  carefully  appropriate  "hands-on"  experience 
into  selected  informational  blocks  of  instruction. 


Need  for  Increased  Student  Contact  Time 


It  is  accepted  that  hands-on  experience  is  a  better  training 
methodology  than  is  lecture  or  observation.  The  AET  is  limited  in  its 
ability  to  provide  all  students  in  a  class  with  sufficient  equipment 
contact  time.  Further,  due  to  cost  and  space  constraints  it  is  not 
feasible  to  acquire  multiple  operational  test  stations  for  training 
purposes.  The  simulators,  however,  do  offer  an  opportunity  to  provide 
more  than  one  or  two  students  with  hands-on  experience  simultaneously. 
The  3-D  simulator  was  configured  to  allow  up  to  four  student  stations 
to  be  operated  from  a  single  instructor  station.  The  2-D  simulator 
was  designed  as  four  stand-alone  components  and  thus  is  theoretically 
capable  of  having  four  student  stations  operate  simultaneously. 

Although  these  capabilities  do  exist,  no  additional  student 
stations  have  been  added  to  the  3-D  simulator.  And,  while  four  compo¬ 
nents  of  the  2-D  simulator  are  available  for  training,  two  of  the 
components  have  not  been  used  to  any  significant  extent  by  the  instruc¬ 
tors.  These  two  components  are  task  specific  and  do  not  directly 
relate  to  the  operation  of  the  ^883  test  station.  While  the  generic 
aspect  of  these  components  should  be  an  asset  to  instruction,  ac¬ 
cording  to  informal  comments  by  instructors  these  components  are  con¬ 
sidered  too  simplistic  to  be  of  any  real  training  value.  (See  also 
instructor  interview  results,  Chapter  V.) 
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In  sum,  then,  simulators  do  offer  the  potential  for  increasing 
student  contact  time  with  equipment  in  a  cost  effective  manner.  In 
the  case  of  the  3-D  simulator,  duplicate  student  stations  could  be 
used.  While  this  approach  is  somewhat  more  expensive  than  the  "part- 
task  trainer"  approach,  it  does  insure  that  each  student  has  access  to 
all  training  protocols  (i.e.,  a  complete  test  station  unit).  The  part- 
task  trainer  approach  employed  by  the  2-D  simulator  is  the  more  cost 
effective  alternative  but  requires  considerable  care  in  the  design 
stage.  For  example,  the  division  of  tasks  among  components  must  be 
made  in  view  of  task  complexity,  training  time  required  on  each  task, 
task  relevancy  to  specific  instructional  blocks,  and  user  acceptance. 

If  these  factors  are  not  adequately  addressed,  it  is  likely  that  some 
system  components  will  not  be  incorporated  effectively  into  the 
training  regime,  and  the  objective  of  increasing  student  contact  time 
with  the  equipment  will  not  be  realized. 


Need  for  Improved  Training 


If  a  primary  objective  of  the  Air  Force  is  to  improve  the 
quality  of  training  delivered,  then  this  study  must  be  considered 
inconclusive.  The  present  evaluation  demonstrates  that  equivalent 
6883  training  can  be  delivered  using  either  a  simulator  or  the  AET. 

It  is  also  reasonable  to  assume  that  simulators  may  offer  some  training 
options  that  are  not  available  with  AETs.  However,  when  simulators 
are  operated  in  the  existing  training  environment  and  used  as  an  alter¬ 
native  to  AETs,  their  potential  training  advantages  cannot  be  realized 
or  measured.  To  properly  determine  if  simulators  are  more  effective 
trainers  than  those  currently  employed,  it  is  necessary  to  design  a 
study  capable  of  testing  one  or  more  of  the  following  hypotheses: 

•  Given  equal  training  time  on  a  simulator  or  AET, 
simulator-trained  airmen  perform  better  than  AET 
trained  airmen. 

•  Simulator-trained  airmen  can  be  trained  to  perform 
at  a  specified  level  of  competence  in  less  time  than 
airmen  trained  on  the  AET. 

•  Simulators  can  increase  the  amount  of  hands-on  train¬ 
ing  time  per  student  within  the  same  training  regime 
currently  used. 

The  first  hypothesis  is  the  one  tested  in  the  present  study. 
However,  the  finding  of  equivalent  training  effectiveness  cannot  be 
considered  conclusive  for  two  reasons.  First,  the  amount  of  training 
actually  delivered  in  the  6883  test  station  block  of  instruction  is 
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very  limited  when  viewed  in  the  context  of  the  total  course.  Second, 
due  to  variations  in  instructor  teaching  styles,  class  sizes,  and 
equipment  availability,  it  is  difficult  to  assume  that  equal  training 
was  provided  to  all  classes.  An  adequate  test  of  the  first  hypothesis 
(given  equal  training  time,  simulator  trained  airmen  perform  better 
than  AET  trained  students)  would  require  an  increase  in  the  amount  of 
the  intervention  (i.e.,  use  of  the  simulator)  and  careful  control  of 
the  material  presented.  Further,  the  performance  test  used  must  be 
validated  in  the  field  ana  shown  to  discriminate  among  various  levels 
of  performance  or  ability,  since  the  relationship  between  school 
performance  and  field  performance  is  unknown  (Orlansky  &  String,  1979). 

The  second  hypothesis  (equivalent  performance  in  less  time 
using  simulators)  has  clear  implications  for  cost  effectiveness  as 
well  as  training  effectiveness.  A  carefully  designed  study  which 
addresses  this  hypothesis  could  also  investigate  the  amount  of  simu¬ 
lator  training  required  to  attain  specified  performance  levels. 

The  third  hypothesis  is  itself  based  on  the  assumption  that 
more  hands-on  training  will  result  in  better  performance.  Given  this 
assumption,  support  for  the  hypothesis  would  demonstrate  that  simula¬ 
tors  are  more  effective  trainers  because  more  airmen  can  be  adequately 
trained  in  a  specified  amount  of  time.  A  test  of  this  hypothesis 
would,  of  course,  require  that  the  option  of  adding  three  student 
stations  to  the  3-D  simulator  be  exercised.  In  the  case  of  the  2-D 
simulator  it  would  be  necessary  to  devise  a  training  protocol  which 
enabled  airmen  to  receive  equivalent  training  by  rotating  through  th» 
four  part-task  trainers. 


Summary 


The  preceding  discussion  illustrates  that  simulators  might  be 
expected  to  fill  a  variety  of  needs  in  the  Air  Force  training  environ¬ 
ments.  They  can  be  more  reliable  in  operation,  less  expensive  to 
acquire  and  maintain,  more  consistent  in  delivery  of  training,  and 
allow  more  student  contact  time  than  actual  equipment.  However,  to 
provide  insight  effectively  into  the  potential  training  and  cost 
advantages  of  simulators  over  AETs  the  purpose(s)  of  introducing  sim¬ 
ulators  into  the  training  environment  must  be  clearly  articulated  and 
the  role  of  the  simulator  must  be  clearly  defined. 

As  the  number  of  objectives  identified  for  simulator  training 
increases,  the  evaluation  design  will  become  more  complex.  Thus,  the 
strength  of  any  study  conducted  will  be  directly  affected  by  the  amount 
of  control  afforded  evaluators  over  key  instructional  variables  (e.g., 
duration  of  training,  content  of  training,  use  of  equipment).  The 
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incorporation  of  simulators  into  the  existing  curricula,  without 
investigating  the  potential  of  such  devices  to  go  beyond  that  framework, 
will  always  leave  the  true  value  of  such  devices  in  doubt. 

Despite  the  numerous  environmental  constraints  on  the  present 
study,  it  is  clear  that  simulators  do  provide  a  cost  effective  alterna¬ 
tive  to  AETs.  Further,  when  introduced  into  existing  curricula,  sim¬ 
ulators  provide  training  equivalent  to  that  obtained  by  using  AETs. 

This  information  alone,  however,  is  insufficient  to  make  knowledgeable 
decisions  regarding  the  use  of  simulators  in  electronic  maintenance 
training.  To  make  such  decisions,  it  is  essential  that  more  complete, 
evaluator  controlled  experimentation,  designed  according  to  clearly 
defined  objectives  and  roles,  be  undertaken. 
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APPENDIX  A 

DATA  COLLECTION  INSTRUMENTS 

Student  Interview 

Instructor  Interview 

Field  Technician  Survey 

Field  Supervisor  Survey 

Results  of  Student  Interview 

Results  of  Instructor  Interview 

Results  of  Technician  Field  Follow-up  Survey 

Results  of  Supervisor  Field  Follow-up  Survey 


SECTION  A-l 


STUDENT  INTERVIEVJ 


NAME  _ 

CLASS  #  _  DATE 


For  each  of  the  questions  below,  please  circle  the  number  that  best 
expresses  your  opinion  on  a  scale  from  1  to  5,  where  1  •  Not  at  all,  and 
5  “  Very  much.  This  information  is  for  use  by  Denver  Research  Institute 
only,  and  will  be  kept  strictly  confidential. 


1.  Did  you  have  sufficient  training 
time  on  the  equipment  in  the 
6883  block? 

2.  Was  your  time  on  the  equipment 
well  utilized? 

3.  Was  there  variety  in  the  training 
on  the  6883  equipment? 

4.  Do  you  have  a  good  understanding 

of  how  to  operate  the  6883  equipment? 

5.  Do  you  feel  comfortable  operating 
the  equipment? 

6.  To  what  extent  did  6883  equipment 
malfunctions  hinder  your  training? 

7.  To  what  extent  did  6883  equipment 
malfunctions  benefit  your  training? 

8.  How  much  of  your  training  time  on 
the  6883  equipment  was  spent  hi 

t  roub leshoot ing  ? 

9.  Would  you  like  to  have  had  more 

t roub l eshoo t ing  experience? 

10.  Do  vou  feel  chat  vou  have  had 
adequate  troubleshoot ing  experience 
for  your  field  assignment? 


Not  at  All  Very  Much 

1  2  3  4  5 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

12  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 


1  2  3  4  5 

1  2  3  4  5 


Form  SI-1 
n.i.te  1  of  2 
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1 


2 


11.  Was  your  training  Instructor  helpful 
in  explaining  the  6883  equipment  and 
its  use? 


3  4  5 


Korn  SI-1 
page  2  of  2 
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SECTION  A- 2 


6883  TRAINING  SIMULATOR  EVALUATION  PROJECT 
INSTRUCTOR  QUESTIONNAIRE 


About  how  long  (in  months)  have  you  been  an  instructor  at  Lowry  AFB? 


Approximately  how  many  times  in  the  past  year  have  you  taught  the 
following  blocks? 

Converter-Flight  Control  _ 

Other  blocks  _ 

If  you  have  had  field  experience  with  Test  Stations,  please  Indicate  the 
length  of  such  experience  and  whether  your  experience  invol'ved  TS 
operation,  TS  maintenance,  or  both  _ 


In  the  next  section,  we  would  like  you  to  rate  how  much  you  would  agree 
with  each  of  the  following  statements  on  a  five  (5)-point  3cale,  where 
1=disagree  strongly  and  5=agree  strongly. 

From  your  general  knowledge  of  and  experience  with  simulated  training, 
do  you  feel  that  simulated  training: 


a. 

is  a  good  idea 

disagree 

3trongly 

1 

2 

3 

4 

agree 

strongly 

5 

b. 

can  be  more  effective  than  actual 

1 

2 

3 

4 

5 

c. 

equipment 

can  provide  equivalent  training  with 

1 

2 

3 

4 

5 

d. 

actual  equipment 

must  be  highly  similar  to  actual 

1 

2 

3 

4 

5 

e. 

equipment  to  be  useful 

can  provide  adequate  training  at  a 

1 

2 

3 

4 

5 

f. 

cost  savings 

allows  for  more  complexity  of  training 

2 

3 

4 

5 

g. 

is  more  reliable  than  actual  equipment 

1 

2 

3 

4 

5 

h. 

teaches  safety  training  better  than 

1 

2 

3 

4 

5 

i. 

actual  equipment 

provides  more  variety  of  training 

1 

2 

3 

4 

5 

J. 

than  actual  equipment 
is  something  you  would  use  as  an 

1 

2 

3 

4 

5 

k. 

integral  part  of  your  teaching  program 
can  replace  actual  equipment  for 

1 

2 

3 

4 

5 

"hands-on"  training 

1 

2 

3 

4 

5 

Form  IQ-2 
page  1  of  4 
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Please  rate  how  important  you  feel  each  of  the  following  factors  is  in 
evaluating  a  simulated  training  device: 


unimportant 


very 

important 


a.  complexity  of  the  equipment  1 

b.  capability  of  the  software  to  meet  1 

STS  and  Air  Force  objectives 

c.  a  lower  cost  of  hardware  and  operating  1 
expenditures  compared  to  actual  equipment 

d.  a  high  degree  of  similarity  of  the  sim-  1 
ulated  equipment  to  the  actual  equipment 

e.  a  savings  in  the  amount  of  time  required  1 
for  training 

f.  the  degree  of  control  of  AF  personnel  1 

over  the  design  of  the  equipment 

g.  the  capability  of  AF  personnel  to  modify  1 
existing  or  create  new  lessons  for  the 
simulated  trainer 


i  h.  mobility  of  the  equipment,  for  versatil-  1 

ity  of  use  in  the  classroom 
i.  reliability  of  performance  of  the  1 

equipment 

J.  ease  of  maintenance  of  the  equipment  1 

k.  ability  to  more  closely  monitor  student  1 
performance  on  the  equipment 

II.  variety  of  material  covered  in  lessons  1 

compared  to  actual  equipment 
m.  ease  of  use  for  training  staff  in  1 

presenting  training  materials. 


2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 


Please  indicate  the  amount  of  experience  you  have  had  with  the  3-D  6383 
Training  Simulator  and  the  2-D  Training  Simulator  by  checking  the 
appropriate  statements  helow: 

For  3-D  For  2-D 


have  heard  about  it,  but  never  actually  used  it 
have  seen  a  demonstration  of  ' t 

have  had  limited  use  of  it,  as  a  reference  for  teaching 

have  used  it  as  a  regular  part  of  my  teaching 

have  been  involved  with  writing  lessons  for  use  on  it 

have  been  involved  with  the  design  and  development 
of  the  unit 

am  a  member  of  the  Special  Projects  3taff 


Form  It'- 2 

pa>;e  2  ot  1 


m 


From  your  experience  with  the  3-D  6883  Training  Simulator,  to  what  extent  would 
you  agree  with  the  following  statements: 


disagree 

agree 

strongly 

strongly 

a. 

the  hardware  is  too  simple  for  it  to 
be  an  effective  training  instrument 

1 

2 

3 

14 

5 

b. 

the  software  (lessons)  is  well-designed 
for  Instruction  purposes 

1 

2 

3 

14 

5 

c. 

the  lessons  meet  STS  and  course 
objectives 

1 

2 

3 

4 

5 

d  • 

students  appear  to  regard  the  simulator 
as  an  actual  Test  Station 

1 

2 

3 

14 

5 

e. 

the  simulator  is  a  better  training 
instrument  than  the  Test  Station 

1 

2 

3 

14 

5 

f. 

I  have  used/will  use  the  Honeywell  sim¬ 
ulated  trainer  as  an  integral  part  of 
my  teaching  program 

1 

2 

3 

4 

5 

From  your  experience  with  the  2-D  6883  Training  Simulator, 

to 

what 

extent  would 

you 

agree  with  the  following  statements: 

disagree 

agree 

strongly 

strongly 

a. 

the  hardware  is  too  simple  for  it  to 
be  an  effective  training  instrument 

1 

2 

3 

4 

5 

b. 

the  software  (lessons)  is  wr ’ \  lesigned 
for  instruction  purposes 

1 

2 

3 

4 

5 

c. 

the  lessons  meet  STS  and  course 
objectives 

1 

2 

3 

4 

5 

d. 

students  appear  to  regard  the  simulator 
as  an  actual  Test  Station 

1 

2 

3 

4 

5 

e. 

the  simulator  is  a  better  training 
instrument  than  the  Test  Station 

1 

2 

3 

4 

5 

f. 

I  have  used/will  use  the  Burtek  sim-  T 

2 

3 

4 

5 

ulated  trainer  as  an  integral  part  of 
my  teaching  program 


Do  you  have  any  additional  comments  to  make  regarding  the  3-D  6883 
Training  Simulator,  its  design  and  use? 


Do  you  have  any  additional  comments  to  make  regarding  the  2-D  6883 
Training  Simulator,  its  design  and  use? 


What  involvement  have  you  had  with  the  Denver  Research  Institute's 
evaluation  of  the  simulated  trainers?  Please  check  any  applicable 
statements. 


a.  _  proctored  the  two-hour  written  test  package 

b.  _  proctored  the  practical  performance  test 

c.  _  assisted  with  the  design  of  the  tests 

d.  _  was  interviewed  regarding  my  teaching  methods  and  course 

material 

e.  _  had  no  involvement  with  the  DRI  evaluation  program  or 

development  of  materials.  (If  no  involvement,  skip  next 
section. ) 


To  what  extent  would  you  say  your  Involvement  with  the  DRI  evaluation 
has  influenced  your  approach  to  teaching?  Check  any  appropriate 
statements. 

a.  _  I  have  become  aware  of  weaknesses  in  my  methods  of  presenting 

block  material. 

b.  _  I  have  become  aware  of  weaknesses  in  the  course  material. 

c.  _  I  have  put  more  emphasis  on  certain  areas  of  the  course 

material. 

d.  _  I  have  made  changes  in  the  Plan  of  Instruction  (POI)  for  the 

block(s)  I  teach. 

e.  _  I  have  requested  changes  in  the  course  material  for  the 

block(s)  I  teach. 

f.  _  I  have  not  been  influenced  in  my  methods  of  teaching  by  the 

the  DRI  evaluation. 

g.  _  I  have  not  seen  any  reason  to  alter  course  materials  due  to 

the  DRI  evaluation. 


Korn  Ii'-2 
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DRI  F— 111  TRAINING  SIMULATOR  EVALUATION  PROJECT 
AVIONICS  TECHNICIAN  SELF-RATING  FORM 


This  form  Is  to  be  used  to  assess  the  field  per¬ 
formance  of  the  technician  named  in  the  box  on 
the  right.  The  information  contained  on  this 
form  will  in  no  way  be  used  to  affect  the 
technician's  personal  or  work  records,  or  en¬ 
hance  or  impede  his/her  career.  This  form 
becomes  the  sole  property  of  the  Denver 
Research  Institute  and  HRL  under  the  terms 
of  Contract  No.  F33615-78-C-0018.  This  infor¬ 
mation  is  being  collected  to  monitor  the  field 
performance  of  those  technicians  whose  train¬ 
ing  performance  was  monitored  at  Lowry  AFB  during 
their  ATC  training  assignment. 


TO  BE  COMPLETED  BY  DRI 


Technician  Last  Name: 

Personal  Security  Number: 
Current  AFB  Assignment: 

Technician  Time  in  Field: 

Training  AET  3-D  2-D 
Testing  AET  3-D  2-D 


1.  What  is  your  current  Test  Station  (TS)  assignment? 


2.  What  previous  TS  assignments  have  you  had  since  you  have  been  in  the  field? 

3.  During  the  6883  Test  Station  (Converter  Flight  Control  Systems)  block  of 
your  ATC  training  at  Lowry  AFB,  were  you  trained  or  tested  on  a  Training 
Simulator? 


(Circle  the  correct  answer) : 

a.  Yes,  trained  on  Simulator 

b.  Yes,  tested  on  Simulator 


c.  Yes,  trained  and  tested  on 

Simulator 

d.  No,  no  contact  with  Simulators 


Don ’ t  know 


On  the  following  pages,  we  would  appreciate  your  help 
on  this  evaluation  project.  Please  answer  the  questions,  to 
the  best  of  your  ability,  using  the  graduated  scale.  The 
questions  relate  to  your  current  working  situation  and  your  ATC 
training  at  Lowry  AFB.  Circle  the  point  on  the  scale  which 
most  accurately  reflects  your  situation  or  opinion. 
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4. 

How  many  hours  of  your  eight-hour  shift 
do  you  spend  actually  operating  the 

TS,  not  including  observation  time? 

0-1  2-3 

I  | 

4-5  6-7 

.  1  1 

8  or  more 

_ I 

5. 

In  operating  the  TS,  how  many  others 
work  at  the  TS  with  you? 

none  1 

1  J. 

2  3 

1  1 

4  or  more 

6. 

How  much  individual  attention  do  you 
receive  on  the  TS  during  OJT? 

none 

i  i 

some 

i  1 

a  lot 

7. 

How  comfortable  do  you  feel  operating 
the  TS  to  which  you  are  currently 
assigned? 

not  at 
all 

1  I 

somewhat 

I  1 

very 

much 

8. 

How  adequate  are  the  TOs  for  trouble¬ 
shooting  the  TS  and  LRUs? 

not  at 
all 

1  1 

somewhat 

1  1 

very 

adequate 

i 

9. 

To  what  extent  does  OJT  address  your 
training  needs  usually? 

not  at 
all 

i  i 

somewhat 

1  1 

very 

much 

10. 

To  what  extent  does  your  current  field 
assignment  meet  your  personal/career 
objectives? 

not  at 
all 

1  1 

somewhat 

1  1 

very 

much 

11. 

How  relevant  was  your  ATC  training  to 
your  current  field  assignment? 

not  at 
all  . 

1  1 

somewhat 

1  1 

very 

much 

12. 

Did  your  ATC  training  adequately  teach 
you  cabling  (of  LRUs  or  adapters)? 

not  at 
all 

I  1 

somewhat 

1  1 

very 

adequate 

13. 

Did  your  ATC  training  adequately  teach 
you  use  of  common  test  equipment 
(PSM-6,  etc.)? 

not  at 
all 

1  1 

somewhat 

1  1 

very 

adequate 

14. 

How  adequate  was  your  ATC  trouble¬ 
shooting  training  for  preparing  you 
for  your  current  assignment? 

not  at 
all 

1  1 

somewhat 

|  | 

very 

much 

15. 

How  Important  was  the  Converter  Flight 
Control  (6883)  block  of  your  ATC 
training  to  your  current  assignment? 

not  at 
all 

1  1 

somewhat 

1  1 

very 

important 

16. 

To  what  extent  does  the  training  you 
missed  at  ATC  (training  for  which  you 
received  deviations)  affect  your  ability 
to  operate  the  TS  in  your  current 
assignment? 

not  at 
all 

1 _ L 

somewhat 

.1  J  ■ 

very 

much 

Form  ATSF-A 
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SIMULATOR  SPECIFIC  PREVIOUS  TRAINING 


17.  Did  your  ATC  training  give  you  not  at  somewhat  very 

adequate  training  on  the  use  all  much 

of  the  patch  panel  as  a  trouble¬ 
shooting  instrument?  I  I  I  I  I 


18.  What  aspects  of  your  ATC  training  do  you  specifically  .use  in  your 
current  field  assignment?  _ 


19 .  What  aspects  of  your  ATC  training  do  you  use  very  little  in  your 
current  field  assignment?  _ 


20.  What  would  you  add  to  the  overall  ATC  training  program  at  Lowry  AFB 
to  better  prepare  avionics  technicians  for  their  field  assignments? 


The  remaining  questions  on  this  questionnaire 
are  directed  to  those  technicians  who  received  some 
training  or  testing  on  a  Training  Simulator  at  Lowry 
AFB.  If  you  circled  responses  a,  b,  or  c  on  question 
3  of  this  questionnaire,  please  complete  the  questions 
to  follow.  If  you  circled  responses  or  je,  you  have 
completed  the  questionnaire  and  should  return  this  form 
to  your  supervisor. 


21. 

To  what  extent  did  you  enjoy  inter¬ 
acting  with  the  Training  Simulator 
during  ATC  training  at  Lowry  AFB? 

not  at 
all 

1 _ L 

some 

_J _ 1 

very 

much 

1 

22. 

To  what  extent  did  the  Training  Simul¬ 
ator  simulate  actual  TS  operating  con¬ 
ditions? 

not  at 
all 

|  | 

some 

1  I 

very 

much 

1 

23. 

Do  you  feel  the  Training  Simulator  was 
probably  a  better  training  instrument 
than  the  actual  TS  for  ATC  training? 

not  at 
all 

1 _ L 

some 

_J _ 1 

very 

much 

_ I 

24. 

Do  you  think  simulated  training  instru¬ 
ments  should  be  used  more  frequently 
during  ATC  training? 

not  at 
all 

1 _ L 

some 

_J _ 1 

very 

much 

i _ l 
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SIMULATOR  SPECIFIC 


25.  In  what  areas  was  the  training  simulator  deficient  or  bothersome: 


response  time 
cabling  procedures 
visual  similarity  to  AET 
operational  similarity  to  AET 
LRU  operation 

TS  trouhlerhooting  capability  ' 

noise  level 

sensitivity  to  dial  settings,  switch  movement  or  solution  selection 
Other _ _ _  _ 


26.  What  did  you  like  or  dislike  about  the  Training  Simulator  as  a  training 
instrument?  _ _ _  _ 


You  have  completed  the  questionnaire.  Please 
return  this  form  to  your  field  supervisor.  Thank  you 
for  your  assistance. 


Form  ATSF-A 
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BACKGROUND  DATA 


SECTION  A-4 


DRI  F-lll  AVIONICS  TECHNICIAN  EVALUATION  PROJECT 


Supervisor  Rating  Form 


This  form  Is  to  be  used  to  assess  the 
field  performance  of  the  technician 
named  In  the  box  on  the  right.  The 
Information  contained  on  this  form 
will  in  no  way  be  used  to  affect  the 
technician's  personal  or  work  records, 
or  enhance  or  Impede  his/her  career 
status.  This  form  becomes  the  sole 
property  of  DRI  and  HRL  under  the 
terms  of  Air  Force  Contract  No. 
F33615-78-C-0018.  This  information 
is  being  collected  to  monitor  the 
field  performance  of  those 
technicians  whose  training  per¬ 
formance  was  monitored  at  Lowry  AFB 
during  their  ATC  assignment. 


TO  BE  COMPLETED  BY  DRI 


Technician  Last  Name: 


Personal  Security  Number: 


Current  AFB  Assignment: 


Technician  Time  in  Field: 


Training  AET  3-D  2-D 
Testing  AET  3-D  2-D 


1.  Name  of  person  rating  technician:  (include  position,  e.g.,  Br.  Chief,  Shop 
Chief,  etc.) 


.  Test  station  technician  is  currently  operating  (e.g.,  6883,  6886,  6863,  etc.): 


.  Other  test  station's  technician  has  operated  during  field  duty: 


A.  Has  technician  been  reassigned  to  duty  other  than  TS  operation? 
Yes _ No _  if  yes,  what  other  duty  and  when? 
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IN  the  following  section,  please  rate  the  technician's  task  performance  on 
a  scale  of  1  to  5,  where  1  signifies  very  poor  or  unsatisfactory  performance, 
2  signifies  poor  performance,  3  signifies  satisfactory  or  fair  performance, 

4  signifies  good  performance,  and  5  signifies  very  good  or  excellent  perfor¬ 
ate.  NA  indicates  this  task  is  not  applicable  in  rating  the  technician  at 
this  time. 


Very 

Poor 

5.  Practices  housekeeping  consistent 

with  the  safety  of  personnel  and 
equipment:  1 

6.  Applies  safety  precautions  when 

maintaining/operating  test  station 
equipment  (for  example,  removes  all 
items  of  metal  such  as  Jewelry, 


watches,  glasses,  etc.):  1 

7.  Uses  technical  manuals  as  a  source  of 

information  for  performing  mainte¬ 
nance  and  Inspection:  1 

8.  Understands  purpose  and  function/ 

operational  concepts  of  test 

stations  in  general:  1 

9.  Understands  purpose  and  function/ 

operational  concepts  of  CENPAC :  1 


10.  Understands  purpose  and  function  of 

Converter  and  Flight  Controls 
test  station  in  particular: 

11.  Understands  theory/signal  flow  of 

particular  LRUs  associated  with  test 


station  listed  above:  1 

12.  Operates  test  station  and  shop  standards 

to  perform  diagnostic  testing:  1 

13.  Operates  test  station  and  shop  standards 

to  troubleshoot  malfunctions:  1 

14.  Operates  test  station  and  shop  standards 

to  perform  maintenance:  l 

15.  Understands  use  of  patch  panel  as 

troubleshooting  instrument:  1 

16.  Performs  cabling  of  LRUs  and  adapters:  1 


Form  SRF-A 
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Very 

Fair  Good 

2  3  4  5  NA 

2  3  4  5  NA 

2  3  4  5  NA 

2  3  4  5  NA 

2  3  4  5  NA 

2  3  4  5  NA 

2  3  4  5  NA 

2  3  4  5  NA 

2  3  4  5  NA 

2  3  4  5  NA 

2  3  4  5  NA 

2  3  4  5  NA 
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ATTITUDE,  INITIATIVE  &  SUPERVISION  TSC  (cont 


Very 

Poor 


Fair 


Very 

Good 


3  4  5  NA 

3  4  5  NA 

3  4  5  NA 

3  4  5  NA 

3  4  5  NA 

3  4  5  NA 

3  4  5  NA 
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17.  Uses  common  test  equipment  to  per¬ 
form  troubleshooting  and  mainte¬ 
nance  of  TS  and  LRUs: 


1  2 


18.  Coordinates  work  with  other  personnel: 


1  2 


19.  Resolves  technical  problems  assigned 
to  him/her: 


1  2 


20.  Works  well  without  significant 
supervision: 


1  2 


21.  Shows  Initiative  in  troubleshooting 
tasks: 


1  2 


22.  Shows  good  work  attitude  toward 
assigned  tasks: 


1  2 


23.  How  would  you  rate  this  technician 

compared  to  other  technicians  at  the 
same  (3,5,7  or  9)  level  of  status? 


1  2 
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SECTION  A-5 


SECTION  A- 6 


INSTRUCTOR  INTERVIEW  RESULTS 


From  your  general  knowledge  of  an  experience  with  simulated  training,  do 
you  feel  that  simulated  training: 


n=9 


e  is  a  good  idea  3*8* 

e  can  be  more  effective  than  aotual  equipment  3.6 

e  can  provide  equivalent  training  with  aotual  equipment  3.0 

e  must  be  highly  similar  to  actual  equipment  to  be  3.3 

useful 

e  can  provide  adequate  training  at  a  cost  savings  4.0 

e  allows  for  more  complexity  of  training  3.8 

e  is  more  reliable  than  actual  equipment  4.6 

e  teaches  safety  training  better  than  aotual  equipment  3.4 

e  provides  more  variety  of  training  than  aotual  4.2 

equipment 

e  is  something  you  would  use  as  an  integral  part  of  4.0 

your  teaching  program 

e  can  replace  actual  equipment  for  "hands-on”  training  2.4 


Please  rate  how  important  you  feel  each  of  the  following  factors  is  in 
evaluating  a  simulated  training  device: 

n=9 

e  complexity  of  the  equipment  3.8 

e  capability  of  the  software  to  meet  STS  and  Air  Force  4.7 
objectives 

e  a  lower  cost  of  hardware  and  operating  expenditures  3-9 
compared  to  actual  equipment 

*1sdisagree  strongly;  5= agree  strongly 


•  a  high  degree  of  similarity  of  the  simulated  equipment  3 .7 
to  the  actual  equipment 

e  a  savings  in  the  amount  of  time  required  for  training  3.7 
e  the  degree  of  control  of  AF  personnel  over  the  design  4.4 


of  the  equipment 

e  the  capability  of  AF  personnel  to  modify  existing  or  4.8 
create  new  lessons  for  the  simulated  trainer 

e  mobility  of  the  equipment,  for  versatility  of  use  in  3.3 
the  classroom 

e  reliability  of  performance  of  the  equipment  4.9 

e  ease  of  maintenance  of  the  equipment  4.6 

e  ability  to  more  closely  monitor  student  performance  3.1 
on  the  equipment 

e  variety  of  material  covered  in  lessons  compare)!  to  4.1 
actual  equipment 

e  ease  of  use  for  training  staff  in  presenting  training  3*9 
materials 


From  your  experience  with  the  3-0  6883  Training  Simulator,  to  what 


extent  would  you  agree  with  the  following  statements: 

n=9 

e  the  hardware  is  too  simple  for  it  to  be  an  effective  2.0 
training  instrument 

e  the  software  (lessons)  is  well  designed  for  3.1 

instruction  purposes 

e  the  lessons  meet  STS  and  course  objectives  3.9 

e  students  appear  to  regard  the  simulator  as  an  actual  3.1 
Test  Station 

e  the  simulator  is  a  better  training  instrument  than  2.8 
the  Test  Station 

e  I  have  used/wlll  use  the  Honeywell  simulated  trainer  * 
as  an  integral  part  of  my  teaching  program 


*Ho  average  response  is  reported  for  this  Item  due  to  the 
ambiguous  wording  of  the  question. 


From  your  experience  with  the  2-D  6883  Training  Simulator,  to  what 


extent  would  you  agree  with  the  following  stateaents: 

n=9 

e  the  hardware  la  too  simple  for  it  to  be  an  effective  2.9 
training  inatruaent 

e  the  aoftware  (leaaona)  ia  well  deaigned  for  3.6 

inatruction  purpoaea 

e  the  leaaona  meet  STS  and  courae  objectivea  3.6 

e  students  appear  to  regard  the  simulator  as  an  actual  2.1 
Test  Station 

e  the  siaulator  ia  a  better  training  instrument  than  2.1 
the  Teat  Station 

e  I  have  used/wlll  uae  the  Burtek  simulated  trainer  — * 

as  an  Integral  part  of  my  teaching  program 


To  what  extent  would  you  aay  your  Involvement  with  the  DRI  evaluation 
has  influenced  your  approach  to  teaching?  Check  any  appropriate 
statements. 


I  have  become  aware  of  weaknesses  in  my  methods  of  presenting 
block  material. 

I  have  become  aware  of  weaknesses  in  the  course  material. 

I  have  put  more  emphasis  on  certain  areas  of  the  course 
material. 

I  have  made  changes  in  the  Plan  of  Instruction  (POI)  for  the 
block(s)  I  teach. 

I  have  requested  changes  in  the  course  material  for  the 
blook(s)  I  teach. 

I  have  not  been  influenced  in  my  methods  of  teaching  by  the 
DRI  evaluation. 

I  have  not  seen  any  reason  to  alter  course  materials  due  to 
the  DRI  evaluation. 


*Saae  as  previous  page 


SECTION  A-7 


TECHNICIAN  FIELD  FOLLOW-UP  SURVEY  RESULTS* 


'scale:  1=not  at  alii  5* very  much 

"not  all  respondents  answered  all  questions 
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SECTION  A- 8 


SUPERVISOR  FIELD  FOLLOW-UP  SURVEY  RESULTS 


Selected  Survey 

Training  Mode 

Questions 

AET 

3-D 

2-D 

Applies  safety  precautions  when  main¬ 
taining/operating  test  station  equip¬ 
ment  (for  example,  removes  all  Items 
of  metal  such  as  jewelry,  watches, 
glasses,  etc.): 

4.6  (n=34)* 

4.7  (n=40) 

4.4  (ns24) 

Understands  purpose  and  function/ 
operational  concepts  of  test  stations 

In  general: 

3.9  (n=31»> 

3.8  (n=4 1 ) 

3.7  (n=23) 

Operates  test  station  and  shop  stan¬ 
dards  to  perform  diagnostic  testing: 

3.5  (n=27) 

3.7  (n=36) 

3.6  (ns22) 

Operates  test  station  and  shop  stan¬ 
dards  to  troubleshoot  malfunctions: 

3.5  (n=29) 

3.5  (n=36) 

3.4  (n*22) 

Operates  test  station  and  shop  stan¬ 
dards  to  perform  maintenance: 

3.5  (n=27) 

3.6  (n=36) 

3.6  (n=21) 

Understands  use  of  patch  panel  as 
troubleshooting  Instrument: 

2.7  (n=1 7) 

3.0  (n=l8) 

3.0  (n=15) 

Performs  cabling  of  LRUs  and  adapters: 

3.9  (n=27) 

4.0  (n=34) 

4.0  (n=22) 

How  would  you  rate  this  technician 
compared  to  other  technicians  at  the 
same  (3,  5,  7  or  9)  level  of  status? 

4.4  (n*34) 

4.4  (n=40) 

4.2  (n*23) 

*not  all  respondents  answered  all  questions 
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DATA  ANALYSIS  NOTES 


As  noted  in  Chapter  V,  an  analysis  of  covariance  framework  was 
adopted  for  this  study.  This  approach  was  expected  to  satisfy  two 
objectives:  increase  the  precision  of  assessment  and  control  for  any 
pre-existing  differences  (bias)  among  experimental  groups.  The  series 
of  analyses  outlined  below  were  developed  to  select  appropriate  covari¬ 
ates  for  these  purposes.  Because  bias  was  believed  to  pose  the  most 
serious  problem  for  data  analysis,  this  issue  was  considered  first  and 
then  supplemented  with  information  designed  to  increase  precision.  In 
brief,  the  procedure  consisted  of  four  steps:  (a)  the  identification 
of  pretraining  measures,  (b)  test  for  bias  in  the  assignment  of 
students  to  training  groups,  (c)  selection  of  indicators  of  bias,  and 
(d)  selection  of  supplementatl  covariates.  Each  of  these  steps  is 
described  more  fully  in  the  following  sections. 


Pretraining  Measures 

A  total  of  17  qualitative  and  quantitative  measures  were 
collected  from  each  student  prior  to  their  participation  in  the  6883 
training  evaluation.  These  variables  included  sex,  standardized  Air 
Force  aptitude  scores,  DRI-administered  preassessment  test  scores  (rf. 
Chapter  IV),  and  achievement  scores  from  previous  instructional  blocks. 
Tables  B-1  through  B-4  summarize  these  variables  as  a  function  of 
eventual  student  training  assignments. 


Test  for  Bias 

A  primary  concern  in  this  study  was  whether  there  were  any 
meaningful  pre-existing  differences  among  students  assigned  to  the 
three  6883  training  modes.  Since  such  a  large  number  of  variables 
could  be  considered  in  this  regard,  typical  univariate  methods  designed 
to  isolate  these  differences  would  capitalize  on  chance.  That  is,  17 
independent  tests  of  bias  might  well  lead  to  spurious  results.  On  the 
other  hand,  it  was  believed  that  a  single  multivariate  analysis 
including  all  17  measures  would  result  in  a  substantial  reduction  in 
the  power  of  the  test  for  bias.  Thus,  a  two-phase  approach  was  adopted 
to  resolve  these  two  problems.  First,  all  17  variables  were  included 
in  a  stepwise  regression  to  predict  assignment  to  training  modes.  In 
the  final  equation,  only  four  variables  were  statistically  significant 
(£<.05)  predictors  of  training  assignment.  These  were  the  Delta  Con¬ 
cealed  Figures  test  score,  the  AF  Administrative  Aptitude  measure,  and 
results  from  the  Electronics  Fundamentals  and  Logic  instructional 
block  tests.  These  four  variables  then  were  identified  as  likely 
indicators  of  bias  in  the  assignment  of  students  to  training. 


Table  B-1 

Males/Females  by  Training  Assignment 


Mode  of  Training 

Students 

Male 

Female 

Total 

AET 

30 

4 

34 

3-D  Simulator 

40 

8 

48 

2-D  Simulator 

40 

6 

46 

TOTAL 


110 


18 


128 


Table  B-2 
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Aptitude  Scores  as  a  Function  of 
6883  Training  Assignments 


Training  Mode 


'reassessment  Measure 

AET 

3-D  Simulator 

2-D  Simulator 

Delta  Concealed 

N 

35 

46 

46 

Figures 

X 

5.60 

4.59 

5.85 

0 

3.28 

3.21 

2.53 

Delta  Reading 

N 

35 

46 

46 

Vocabulary 

X 

26.5 

27.5 

24.3 

0 

7.1 

7.2 

6.5 

N  35  45  46 

Ship  Destination  Y  35.5  34.9  36.8 

a  9.0  9.4  7.5 


Table  B-4 


Prior  Block  Achievement  Scores  as  a 
Function  of  6883  Training  Assignments 


Training  Mode 


Instructional  Block 

AET 

3-D  Simulator 

2-D  Simulator 

Fundamentals  of 

N 

35 

47 

46 

Electronics 

X 

89.8 

87.0 

87.4 

a 

6.6 

6.2 

6.0 

Introduction  to  AGE 

N 

35 

48 

46 

Principles 

X 

89.6 

88.6 

85.5 

a 

8.1 

8.6 

9.0 

N 

35 

48 

46 

CENPAC 

X 

83.1 

83.4 

84.0 

0 

9.7 

10.7 

9.0 

N 

35 

48 

46 

DATAC 

X 

84.2 

86.6 

84.0 

a 

9.7 

7.9 

10.7 

Data  Logic 

N 

35 

48 

46 

Analysis 

X 

85.0 

87.1 

85.7 

0 

11.6 

9.9 

9.0 

N 

35 

48 

46 

CATE 

X 

88.6 

85.7 

86.0 

a 

8.7 

10.0 

8.3 

Navigation  and 

N 

35 

48 

46 

Weapons 

X 

85.6 

85.8 

86.8 

a 

10.9 

9.9 

9.3 

Electronic 

N 

35 

48 

46 

Systems 

X 

88.4 

86.7 

85.8 

0 

11.6 

9.5 

10.9 

The  second  phase  involved  submitting  these  variables  to  a 
multivariate  analysis  of  variance  (MAN OVA)  to  determine  whether  or  not 
there  were  any  pre-existing  differences  among  the  three  training  groups. 
This  analysis  revealed  a  significant  bias  effect  (F  (8,218)  =  2.63, 

£  =  .01) . 


Selection  of  Bias  Indicators 


While  the  previous  analysis  suggested  that  some  bias  existed 
in  experimental  groups  and  which  variables  might  be  used  to  reflect 
that  bias,  a  further  step  was  necessary  in  order  to  finalize  the 
selection  of  bias  indicators.  The  question  addressed  at  this  point 
was:  Are  all  four  of  these  indicators  necessary  to  appropriately 
reflect  bias?  An  extension  of  the  MANOVA  procedure  used  in  the 
previous  section  was  used  to  answer  this  question.  First,  the 
dependent  variable  in  the  MANOVA  with  the  greatest  univariate  F-ratio 
as  a  bias  indicator  was  now  considered  as  a  covariate,  thus  removing 
its  contribution  from  the  main  effect.  Then,  a  multivariate  analysis 
of  covariance  (MANCOVA)  was  performed.  Because  the  main  effect  was 
still  significant,  another  variable  was  shifted  from  the  dependent 
variable  list  to  the  covariate  list  as  before.  After  three  variables 
had  been  treated  as  covariates  in  this  manner,  the  main  effect  of  the 
remaining  variable  (Administrative  Aptitude)  was  not  found  to  be  sig¬ 
nificant.  The  other  three  variables  (Delta  Concealed  Figures,  Elec¬ 
tronics  Fundamentals,  Logic)  were  then  selected  as  final  indicators  of 
bias. 


Supplemental  Covariates 

The  gain  in  precision  from  the  use  of  a  eovariance  adjustment 
depends  upon  the  degree  of  correlation  between  the  covariate  and  the 
dependent  variable.  Thus,  while  bias  indicators  had  been  identified, 
a  second  objective  was  to  select  covariates  that  were  strongly  related 
to  each  measure  of  performance  or  indicator  of  subsequent  job  perform¬ 
ance.  Although  similar  principles  were  applied  to  each  type  of 
dependent  variable,  the  analysis  results  have  been  presented  in  the 
following  sections. 

Performance  measures.  Three  performance  measures  were  recorded 
in  this  experiment:  Hands-On  Troubleshooting  Test  scores,  Paper-and- 
Peneil  Troubleshooting  Test  scores,  and  Converter/Flight  Control  (6883) 
block  test  scores.  For  each  variable,  covariates  were  selected  by 
using  a  regression  analysis  in  which  all  three  indicators  of  bias  were 
entered  first  into  the  regression  equation.  For  example,  an  equation 
was  computed  for  predicting  Hands-On  Test  scores  with  Delta  Concealed 
Figures,  Electronics  Fundamentals,  and  Logic  scores  entered  first  in  a 
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stepwise  fashion.  Additional  variables ,  from  the  original  set  of  17 
measures,  were  then  added  to  the  equation  until  a  total  of  five  vari¬ 
ables  were  included.  From  these  five,  only  those  found  to  be  signif¬ 
icant  predictors  of  performance  were  selected  as  final  covariates  for 
use  in  the  ANCOVAs  reported  in  Chapter  V.  It  was  possible  that  indi¬ 
cators  of  bias  were  not  strongly  related  to  Hands-On  Test  results  and 
these  were  not  included  in  subsequent  analyses.  In  this  example,  only 
the  Delta  Concealed  Figures  variable  was  retained  as  a  covariate  from 
the  bias  indicators  and  two  additional  variables,  General  Aptitude 
Test  scores  and  CATE  instructional  block  scores,  were  selected  to 
supplement  the  covariance  approach. 

Table  B-5  shows  the  final  covariates  selected  for  use  in  each 
ANCOVA  involving  performance  measures.  In  each  case,  indicators  of 
bias  were  selected  as  covariates  only  if  the  relationship  between  the 
indicator  and  the  dependent  variable  was  sufficiently  strong. 

Job-related  measures.  This  procedure  to  select  covariates  was 
also  applied  to  two  types  of  job  related  measures:  the  Projected  Job 
Proficiency  Test  and  subsequent  block  test  results.  The  outcome  of 
this  procedure  has  also  been  included  in  Table  B-5.  A  multivariate 
approach  to  subsequent  block  scores  would  have  been  preferred;  however 
such  an  analysis  was  not  possible  since  students  were  not  all  assigned 
to  the  same  instructional  track. 


Table  B-5 


List  of  Covariates  Selected  for  Dependent 
Variables  Used  in  This  Study 


Dependent  Variable 

Covariates  Selected 

Performance  Measures 

Hands-On  Troubleshooting 

Delta  Concealed  Figures 
General  Aptitude 

CATE  Block 

Paper-and-Pencil  Troubleshooting 

Electronics  Fundamentals 
Electronic  Aptitude 
Mechanical  Aptitude 

Converter/Flight  Control  Block 

Administrative  Aptitude 
Electronics  Fundamentals 
CENPAC  Block 

Job-Related  Measures 

Project  Job  Proficiency 

Electronics  Fundamentals 
DATAC  Block 

Ship  Destination 

Computer  Test  Station  Block 

Electronics  Fundamentals 

Sex 

Navigation  Block 

Attitude  and  Rate  TS  Block 

CENPAC  Block 

Ship  Destination 

Displays  TS  Block 

Electronics  Fundamentals 
Logic  Block 

A GE  Principles  Block 

Video  TS  Block 

Navigation  Block 

Electronics  System 

Radar-Transmitter  Modulator 

TS  Block 

Delta  Concealed  Figures 

CATE  Block 

CENPAC  Block 
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NOISB  LEVEL  ANALYSIS  NOTES 


Since  it  was  not  possible  to  remove  the  background  noise 
(central  air  conditioning)  from  the  training  environments  to  obtain 
equipment-only  noise  level  readings,  calculations  were  made  to  subtract 
the  background-only  noise  from  the  overall  equipment  and  background 
noise  levels.  The  standard  logarithmic  decibel  scale  formula  used 
was: 


SPLfc0fcai  =  SPL-|  +  SPL2 

=  10  log10  (1_0SPLj  +  10  SPL2) 

10  10 

Where  SPL  =  sound  pressure  level  or  dBa 
SPL-)  =  equipment-only 
SPL2  =  background  noise 

Working  with  the  overall  dBa  readings  and  the  background 
readings,  equipment-only  readings  were  obtained  for  each  of  the 
distance  measurements  for  each  of  the  training  devices.  For  example, 
if: 

SPLtotal  *  73.9  dBa 
SPLbackground  =  ^5  dBa 

then: 

SPLequipment-only  =  10  logio  73.9  -  10  log  10  65 

10  10 


=  10 

log  10 

(107 

.39  - 

106. 5) 

=  10 

log  10 

(101 

•  39  _ 

10*5)  106 

=  10 

log  (24.5  - 

3.16) 

106 

=  10 

(7.32) 

=  73. 

.2  dBa 

Table  C— 1  shows  the  relationship  between  addition  or  subtrac¬ 
tion  of  noise  levels.  For  example,  if  the  difference  between  two  given 
noise  levels  is  5  dBa,  the  total  level  of  both  noises  combined  is  only 
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1.2  dBa  higher  than  the  larger  of  the  two  levels.  Thus,  in  the  example 
above,  removing  65  dBa  from  an  overall  level  of  73.9  dBa  results  in  a 
difference  of  only  .7  dBa  for  the  equipment-only  noise  level. 


Table  C-1 

Addition  and  Subtraction  of  Sound  Pressure  Levels 
(Rank  in  Order  from  High  to  Low) 


Excess  in  dB 

Add  to  higher 

0 

3.0 

1 

2.6 

2 

2.1 

3 

1.8 

4 

1.5 

5 

1.2 

6 

1.0 

7 

0.8 

8 

0.6 

9 

0.6 

10 

0.4 

The  results  of  the  noise  level  comparisons  indicated,  based  on 
Table  C-2,  that  instructor-student  communication  will  be  greatly 
influenced  by  the  distance  between  them  and  will  require  various  vocal 
intensities  for  effective  communication. 
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Table  C-2 


Maximum  Speech  Interference  Levels  (in  dBa)  for  Reliable 
Communication  at  Various  Distances  and  Vocal  Efforts 


Distance  (feet) 

Vocal  Effort 

Normal 

Raised 

Loud 

Shout 

0.5 

76 

82 

88 

94 

1 

70 

76 

82 

88 

2 

64 

70 

76 

82 

4 

58 

64 

70 

76 

8 

52 

58 

64 

70 

16 

46 

52 

58 

64 

32 

40 

46 

52 

68 

l 
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NOTES  ON  COST  ANALYSIS 


A.  Replacement  cost  of  space  estimate  was  taken  from  the  F-16  cost  study 
(Table  8).*  The  estimate  of  $l6.27/square  foot  represents  the  amount 
the  building  inventory  at  Lowry  AFB  would  be  reduced  at  the  end  of 

1977  if  a  training  facility  were  scrapped. 

B.  The  supplemental  furnishings  estimate  for  laboratory  space  was  based 
on  the  approximate  cost  of  items  used  in  the  laboratories. 

C.  The  acquisition  cost  estimates  for  the  AET  were  guided  by  the  esti¬ 
mates  made  for  the  F-16  trainer.  The  acquisition  management  cost 
for  the  36th  unit  of  production  was  .82  percent  of  the  unit  cost 
(Table  5).*  Our  estimate  of  .75  percent  was  more  conservative 
because  the  technology  of  the  F-111  is  not  as  sophisticated  as  that 
of  the  F-16,  even  though  the  F-111  station  represents  the  eleventh 
unit  produced. 

D.  Sustaining  Investment  estimate  was  guided  by  the  estimates  made  for 
the  F-16  trainer  which,  in  turn,  were  derived  from  Air  Force 
Logistic  Command  experience.  The  Sustaining  Investment  required  was 
found  to  be  approximately  12.5  percent  of  the  unit  cost  each  year 
(Table  5).*  Our  estimate  of  $195,400  per  year  was  10  percent  of  the 
unit  cost  of  the  test  station.  The  unit  cost  was  inflated  79.5 
percent  to  reflect  wholesale  commodity  price  changes  from  1972  to 

1978  (source:  Economic  Indicators,  U.S.  Government  Printing  Office, 
Washington,  DC,  1979).  No  estimate  was  made  for  logistics  costs 
associated  with  the  spares  inventory. 

E.  The  personnel  cost  of  maintenance  on  the  6883  AET  was  based  on 
records  of  actual  hours  spent  on  repair  from  1/1/79  to  7/30/79. 

(BLIS  Report,  Logistics  Branch,  Lowry  AFB).  The  time  spent  during 
this  period  was  assumed  to  be  proportional  to  the  yearly  maintenance 
requirement.  The  grade  level  for  costing  purposes  was  estimated  to 
be  E-5  for  the  estimated  672  hours/year.  A  BLIS  Report  for  this 


•The  F-16  study  refers  to  a  draft  technical  report,  "Life  Cycle  Cost 
Estimation  of  Simulated  vs.  Actual  Equipment  Maintenance  Training  for 
the  F-16  Avionics  Intermediate  Shop."  This  report  was  prepared  by  the 
AFHRL  Advanced  Systems  Division,  Wright-Patterson  AFB,  Ohio  (T. 
Eggemeier,  March  1979).  The  report  (unpaged)  was  a  substantial  resource 
for  cost  data  appropriate  to  training  at  Lowry  AFB,  and  the  equipment 
cost  model  employed  provided  substantial  insight  for  estimates  related 
to  acquisition  management  costs  and  the  logistical  costs  including 
Sustaining  Investment  for  weapons  subsystems. 
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figure  was  not  included  in  the  table  because  when  salary  changes  and 
adjustments  to  1978  dollars  are  made,  the  result  is  not  substantially 
different  from  the  original  estimate  of  $3, 620/year.  1980  showed  a 

total  of  641.6  hours  were  spent  on  repair  of  the  same  equipment. 

F.  Since  no  student  materials  are  required  for  6883  laboratory  work, 
this  cost  was  assumed  to  be  zero. 

G.  Updating  laboratory  exercises  was  assumed  to  require  one-sixth  of  an 
instructor  year.  Updating  represents  such  activities  as  installing 
"faults"  in  the  AET  to  facilitate  "manual"  troubleshooting 
exercises. 

H.  Instructor  contact  time  for  practical  training  on  the  6883  test 
station  was  estimated  at  540  hours  per  year  (based  on  approximately 
6  hours  a  day  of  actual  student-instructor  classroom  interaction; 
three  days  of  instruction  per  class;  30  classes  per  year).  The 
estimate  was  increased  to  one-third  of  a  year  (693  hours)  to  more 
accurately  account  for  the  fact  that  one  full-time  instructor  would 
actually  be  assigned  to  teach  6883  theory  and  practice. 

I.  The  overhead  or  burden  associated  with  maintaining  each  involved 
instructor  for  a  specified  time  period  was  estimated  by  apportioning 
the  salary  of  the  course  manager  (GS-12),  the  course  supervisor  (E- 
7),  and  the  instructor's  supervisor  (E-6)  to  management  and 
administration  of  24  instructors  for  the  same  time  period.  (Salary 
schedule  source:  AFR-173-10,  1978). 

J.  Miscellaneous  student  support  costs  for  students  in  residence  at 
Lowry  AFB  were  estimated  at  $79.25  per  student  per  week;  540  student 
days  represent  2.077  years,  assuming  260  working  days  per  year 
(2.077  x  52  x  $79.25  =  $8,550). 

K.  Estimate  based  on  AFHRL  project  records:  1056.5  hours  (military)  and 
2357.5  hours  (civilian)  prior  to  installation  of  the  simulator;  cost 
factors  for  military  and  civilian  personnel  for  AFHRL  were  assumed 
to  be  $12/hour  and  $l4/hour  over  the  18-month  period. 

L.  Estimate  based  on  AFHRL  project  records:  1482  hours  (military)  and 
2374  hours  (civilian)  for  support  and  installation.  Same  cost  fac¬ 
tors  as  used  for  (K),  above. 

M.  Sustaining  Investment  for  the  6883  3-D  simulator  was  based  on  the 
cost  of  maintenance  contracts  in  force  from  June  1978  until  November 
1979,  and  a  contract  being  negotiated  for  1980-1981. 

N.  2-D  contract  value  was  $393»900.  Deflated  25.5  percent  to  adjust 
December  1979  dollars  to  1978  dollars  for  capital  equipment  (Source: 
Economic  Indicators,  U.S.  Government  Printing  Office,  Washington, 

DC,  1980). 
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O.  Estimate  for  specification  and  acquisition  management  computed  as 
one-quarter  time  for  one  year  (520  hours  civilian)  at  $l4/hour.  It 
should  be  noted  that  this  time  investment  is  substantially  less  than 
that  for  the  3-D  simulator  since  specifications  already  developed 
formed  a  large  part  of  the  specifications  for  the  2-D  simulator. 

P.  Estimate  for  support  and  installation  of:  430  hours  (military) 
figured  at  $  12/hour  was  provided  by  the  AFHRL  2-D  project  monitors. 
Additional  time  reported  by  the  ATC  special  projects  staff  include: 
6.75  months  of  E-7;  6.75  of  E— 5 ;  and  4  months  of  E-3. 

Q.  Sustaining  Investment  for  the  6883  2-D  simulator  was  initially  based 
on  the  cost  of  maintenance  contracts  in  force  from  March  -  September 
1981  ($40,600).  This  amount  was  deflated  by  21  percent  ($33,550)  to 
reflect  to  1978  wage  levels.  For  subsequent  years  the  7-month  con¬ 
tract  was  used  to  project  a  yearly  maintenance  cost  of  $57,100. 

This  amount  was  also  deflated  by  21  percent  ($47,190)  to  reflect 
1978  wage  levels.  Note,  however,  that  figures  for  actual  expendi¬ 
tures  provided  by  the  AFHRL  2-D  project  monitor  differed 
substantially  from  these  estimates.  Based  on  actual  contract 
expenditures,  sustaining  cost  figures  were  estimated  to  be  only 
$5,200  in  the  investment  year  and  $2,000  for  subsequent  years. 
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